Using csvkit (csvgrep, csvcut, csvsort, csvstat) to quickly search and analyze 260K (60MB) Chicago crime records

by dann
macOS ◆ xterm-256color ◆ bash 850 views

First, download the 2017 Chicago Crime data using curl

(and save it to /tmp/chicagocrimes.csv

curl -o /tmp/chicagocrimes.csv \
    'https://data.cityofchicago.org/api/views/d62x-nvdr/rows.csv?accessType=DOWNLOAD'

Use csvcut to select a few columns

$ cat /tmp/chicagocrimes.csv \
 | csvcut -c 'Date,Primary Type,Description' \
 | head -n 10

Result:

Date,Primary Type,Description
09/21/2017 12:15:00 AM,DECEPTIVE PRACTICE,COUNTERFEIT CHECK
10/12/2017 07:14:00 PM,CRIMINAL DAMAGE,TO CITY OF CHICAGO PROPERTY
10/30/2017 11:52:00 AM,OTHER OFFENSE,SEX OFFENDER: FAIL REG NEW ADD
09/29/2017 06:45:00 PM,DECEPTIVE PRACTICE,BOGUS CHECK
12/01/2017 06:20:00 AM,CRIM SEXUAL ASSAULT,AGGRAVATED: OTHER
06/08/2017 04:02:00 PM,WEAPONS VIOLATION,UNLAWFUL POSS OF HANDGUN
06/25/2017 07:00:00 PM,ROBBERY,ARMED: HANDGUN
09/06/2017 01:30:00 PM,THEFT,OVER $500
11/23/2017 03:14:00 PM,ASSAULT,AGGRAVATED: HANDGUN

Using csvgrep to search for gun-related crimes

$ cat /tmp/chicagocrimes.csv \
  | csvcut -c 'Date,Primary Type,Description' \
  | csvgrep -c Description -m 'GUN' \
  | head -n 10

Result:

Date,Primary Type,Description
06/08/2017 04:02:00 PM,WEAPONS VIOLATION,UNLAWFUL POSS OF HANDGUN
06/25/2017 07:00:00 PM,ROBBERY,ARMED: HANDGUN
11/23/2017 03:14:00 PM,ASSAULT,AGGRAVATED: HANDGUN
12/11/2017 07:15:00 PM,ROBBERY,ARMED: HANDGUN
12/12/2017 06:00:00 PM,ROBBERY,ARMED: HANDGUN
08/30/2017 01:04:00 AM,BATTERY,AGGRAVATED: HANDGUN
12/06/2017 09:15:00 AM,ROBBERY,ARMED: HANDGUN
12/12/2017 02:04:00 AM,ROBBERY,ATTEMPT: ARMED-HANDGUN
01/16/2017 12:30:00 AM,ASSAULT,AGGRAVATED: HANDGUN

Using csvsort to get the 10 most recent gun-related crimes of 2017:

$ cat /tmp/chicagocrimes.csv  \
    | csvcut -c 'Date,Primary Type,Description' \
    | csvgrep -c Description -m 'GUN' \
    | csvsort -c Date -r \
    | head -n 10

Result:

Date,Primary Type,Description
2017-12-31T23:45:00,BATTERY,AGGRAVATED: HANDGUN
2017-12-31T23:28:00,WEAPONS VIOLATION,UNLAWFUL POSS OF HANDGUN
2017-12-31T23:20:00,ASSAULT,AGGRAVATED: HANDGUN
2017-12-31T23:15:00,WEAPONS VIOLATION,UNLAWFUL POSS OF HANDGUN
2017-12-31T22:51:00,ASSAULT,AGGRAVATED: HANDGUN
2017-12-31T22:45:00,BATTERY,AGGRAVATED: HANDGUN
2017-12-31T22:35:00,WEAPONS VIOLATION,UNLAWFUL POSS OF HANDGUN
2017-12-31T22:05:00,ASSAULT,AGGRAVATED: HANDGUN
2017-12-31T22:03:00,ROBBERY,ARMED: HANDGUN

Getting a frequency count of gun-related crimes, by “Primary Type”:

$ cat /tmp/chicagocrimes.csv  \
    | csvcut -c 'Primary Type,Description' \
    | csvgrep -c Description -m 'GUN' \
    | csvstat -c 'Primary Type' --freq

Result:

{ "ROBBERY": 4952, 
"WEAPONS VIOLATION": 3928, 
"ASSAULT": 2969, 
"BATTERY": 2055, 
"OTHER OFFENSE": 497 }