"Py" Kind of Town

Python in Chicago

@tanyaschlusser—DuPage Python Meetup

• slides: bit.ly/py-town-dupage
• github.com/tanyaschlusser/py-kind-of-town
• hit [space] or [right-arrow] to advance slides
• hit [escape] then [right-arrow] for speed view

Who are you?

(from Meetup data)

Membership stats (on 29 Dec 2018)

• 147 people in the DuPage Python group
• 88% — at least one other Python group
• 50% — at least one Data Science group
• 33% — at least one infrastructure group
• 27% — at least one R group
• 23% — at least one front end group

Intersections (top 10 tags)

Without R may be clearer

That was an upsetplot

• Shows multiple set insersections
• Lex, Gehlenborg, Strobelt, Romain, Pfister (2014)
• Primary implementation: in the Caleydo suite
• R implementation: UpSetR
• Python implementation UpSetPlot (unofficial)

Back story: 6-way Venn Diagram

This figure (Nature, 2012) so offended the authors 😱 that they published a new method in 2014.

Banana redo

Here's the UpSet banana plot.
(With most frequent intersections.)

Description

(from their website)

The dots denote intersections.

Here's their online viewer

Pnemonic: UpSet

...Venn Diagrams shouldn't make you up-set!

Meetup API

How did I get the data?

• REST API that returns JSON-formatted text
• Well-documented meetup.com/meetup_api/
• As of December 2018 you need a free API key
• Many APIs, very detailed; paid users get more

DuPage Python's interests are roughly

(overlapping)

• ½ analytics
• ⅓ infrastructure
• ¼ web

And regional employers?

based on a single-day job board snapshot …

• Free ones
  StackOverflow (RSS) (great!)
  GitHub API (meh)
• Others not used (need a developer key)
  Dice, LinkedIn, Google indexing

Stackoverflow Jobs

(Chicago 5 Jan 2019)

• 96 postings in the Chicago area right now
• 27% Director / Manager / Project Manager jobs
• 18% Backend jobs
• 17% Other developer jobs
• 11% Frontend jobs
• 11% Security jobs
• 7% DevOps jobs
• 7% Full Stack jobs
• 1% Data Science jobs 😭

By employer

• 96 postings in the Chicago area
• 54% Other companies
• 19% JP Morgan Chase
• 6% Uline
• 5% Discover
• 4% Infutor Data Solutions
• 4% Hays US
• 4% NextCapital
• 3% Critical Mass

Tagged language(s) — there's overlap

• 96 postings in the Chicago area
• 36% tagged Java
• 35% tagged Python
• 23% tagged SQL
• 22% tagged Javscript
• 9% tagged C++
• 5% tagged Ruby

Intesections

This uses data from a different date than the last 3 slides

Intersections with just languages

This uses data from a different date than the last 3 text slides

GitHub Jobs (5 Jan 2019)

• 180 U.S. postings
• 22% Other job categories
• 21% Manager/Director/Project Manager jobs
• 14% Front End jobs
• 14% Back End jobs
• 13% Full stack jobs
• 9% DevOps jobs
• 6% Data Science jobs
• 2% Security jobs

GitHub by language (+ overlap)

• 180 U.S. postings
• 40% tagged SQL
• 39% tagged Javascript
• 37% tagged Python
• 26% tagged Scala
• 16% tagged Java
• 9% tagged C++

Conclusion

(caveat: snapshot on a single day)

• 4x-10x less demand for data science than web
• Python is ubiquitous, and usually requested
  in addition to another language

SEC API

Start here: sec.gov/developer

• The EDGAR API
• Electronic Data Gathering, Analysis, and Retrieval system
  sec.gov/edgar/searchedgar/accessing-edgar-data.htm

SEC EDGAR data

(no APIkey)

• Max 10 queries/sec; else possibly block IP
• Press releases, legal announcements
• SEC forms/filings by public companies/investment firms since 1994
• Use regex to pull the parts you want

Options

• RSS feed (recent SEC publications; filings)
  sec.gov/about/secrss.shtml
• Daily / quarterly index files
  (fixed-width text linking to each doc)
• Archive files (zipped)

… produces this:

S1 filing data from:

sec.gov/cgi-bin/srch-edgar

The index files

url = "https://www.sec.gov/Archives/edgar/daily-index/2018/QTR4/form.20181231.idx"
response = requests.get(url)
print(response.text[:2000])

Description:           Daily Index of EDGAR Dissemination Feed by Form Type
Last Data Received:    Dec 31, 2018
Comments:              webmaster@sec.gov
Anonymous FTP:         ftp://ftp.sec.gov/edgar/
 
 
 
 
Form Type   Company Name                                                  CIK
      Date Filed  File Name
---------------------------------------------------------------------------------------------------------------------------------------------
1-A/A       American Diversified Energy LLC                               1689500     20181231    edgar/data/1689500/0001683168-18-003825.txt         
1-A/A       UMBRA Technologies (US) Inc.                                  1742808     20181231    edgar/data/1742808/0001742808-18-000004.txt         
1-SA        Nuvus Gro Corp                                                1671132     20181231    edgar/data/1671132/0001079974-18-000687.txt         
1-SA        Red Fish Properties, Inc.                                     1703429     20181231    edgar/data/1703429/0001703429-18-000003.txt         
1-U         FUNDRISE REAL ESTATE INVESTMENT TRUST, LLC                    1645583     20181231    edgar/data/1645583/0001144204-18-066698.txt         
10-D        BBCMS Mortgage Trust 2017-C1                                  1696707     20181231    edgar/data/1696707/0001056404-18-009029.txt         
10-D        Benchmark 2018-B1 Mortgage Trust                              1722194     20181231    edgar/data/1722194/0001056404-18-009045.txt         
10-D        Benchmark 2018-B4 Mortgage Trust                              1740849     20181231    edgar/data/1740849/0001056404-18-009059.txt         
10-D        CD 2018-CD7 Mortgage Trust                                    1746406     20181231    edgar/data/1746406/0001056404-18-009057.txt         
            

Some fixed width, some delimited. Parse to get the document URLs you care about. Prefix is: https://www.sec.gov/Archives.

The Daily Archives

• /edgar/Feed
  Every literal file uploaded for the day, tarred and zipped in a directory.
• /edgar/Oldloads
  Every literal file of the day, concatenated into a gargantuan zipped file.

SEC Archive data is:

• Great for detailed research of every single SEC form filed since 1994.
• A slight pain because so many of the forms are kind of freestyle text.
• When the form's XBRL you just write the parser once and done!

SEC paid data

EDGAR®Online DataFied API

• developer.edgar-online.com/member/register
  for API key. Max 2 queries/sec
• Companies Metadata:
  SIC category, location, symbol
• Core Financials:
  Balance Sheet, Cash Flow, Income Statement

The 'hackathon' subscription

https://developer.edgar-online.com/docs/v1#nav12

Chicago public companies

Core Financials API

Free concept groups group related terms together:

• Balance Sheet
• Cash Flow
• Income Statement
• Valuation, Profitability, Leverage mini-ratios
  (mini as in less detail than paid version)

Pandas can then go to Excel …

Mini-project?

Keeping with the Chicago theme, try looking at Sears and its competitors

• Identify competitors (same SIC code)
• Look up debt-to-equity ratio
• Compare across competitors

Sears competitors

Some of the results

Plotted

Whaa? 😕

Some articles say Burlington is doing OK. We know Bonton went bankrupt. Analysts are panning Dillard's. Are these numbers wild or do I just not understand retail? (seems like the latter)

Current Ratio

Net Income Per Employee

Conclusion

• All you need is an API key, requests, and some knowledge of Pandas and JSON to get lots of financial data.
• If you want to read the SEC filings you'll also need some LXML skills.
  (but not too much see my stuff)

Thank you

• slides: bit.ly/py-town-dupage
• github.com/tanyaschlusser/py-kind-of-town
• Deck is made using Hakim Hattab's incredible Reveal.js