Skip to content

fitbitparsed_csv

This data stream handles Fitbit sensor data downloaded using the Fitbit Web API, parsed, and stored in a CSV file. Please note that RAPIDS cannot query the API directly; you need to use other available tools or implement your own. Once you have your parsed sensor data in a CSV file, RAPIDS can process it.

What is the difference between JSON and plain data streams

Most people will only need fitbitjson_* because they downloaded and stored their data directly from Fitbit’s API. However, if, for some reason, you don’t have access to that JSON data and instead only have the parsed data (columns and rows), you can use this data stream.

Warning

The CSV files have to use , as separator, \ as escape character (do not escape " with ""), and wrap any string columns with ".

Example of a valid CSV file
"device_id","heartrate","heartrate_zone","local_date_time","timestamp"
"a748ee1a-1d0b-4ae9-9074-279a2b6ba524",69,"outofrange","2020-04-23 00:00:00",0
"a748ee1a-1d0b-4ae9-9074-279a2b6ba524",69,"outofrange","2020-04-23 00:01:00",0
"a748ee1a-1d0b-4ae9-9074-279a2b6ba524",67,"outofrange","2020-04-23 00:02:00",0
"a748ee1a-1d0b-4ae9-9074-279a2b6ba524",69,"outofrange","2020-04-23 00:03:00",0

Container

The container should be a CSV file per sensor, each containing all participants’ data.

The script to connect and download data from this container is at:

src/data/streams/fitbitparsed_csv/container.R

Format

The format.yaml maps and transforms columns in your raw data stream to the mandatory columns RAPIDS needs for Fitbit sensors. This file is at:

src/data/streams/fitbitparsed_mysql/format.yaml

If you want to use this stream with your data, modify every sensor in format.yaml to map all columns except TIMESTAMP in [RAPIDS_COLUMN_MAPPINGS] to your raw data column names.

All columns are mandatory; however, all except device_id and local_date_time can be empty if you don’t have that data. Just have in mind that some features will be empty if some of these columns are empty.

FITBIT_HEARTRATE_SUMMARY

RAPIDS_COLUMN_MAPPINGS

RAPIDS column Stream column
TIMESTAMP FLAG_TO_MUTATE
LOCAL_DATE_TIME local_date_time
DEVICE_ID device_id
HEARTRATE_DAILY_RESTINGHR heartrate_daily_restinghr
HEARTRATE_DAILY_CALORIESOUTOFRANGE heartrate_daily_caloriesoutofrange
HEARTRATE_DAILY_CALORIESFATBURN heartrate_daily_caloriesfatburn
HEARTRATE_DAILY_CALORIESCARDIO heartrate_daily_caloriescardio
HEARTRATE_DAILY_CALORIESPEAK heartrate_daily_caloriespeak

MUTATION

  • COLUMN_MAPPINGS (None)

  • SCRIPTS

    src/data/streams/mutations/fitbit/add_zero_timestamp.py
    

Note

add_zero_timestamp adds an all-zero column called timestamp that will be filled in later in the pipeline by readable_time.R converting LOCAL_DATE_TIME to a unix timestamp taking into account single or multiple time zones.

Example of the raw data RAPIDS expects for this data stream
device_id local_date_time heartrate_daily_restinghr heartrate_daily_caloriesoutofrange heartrate_daily_caloriesfatburn heartrate_daily_caloriescardio heartrate_daily_caloriespeak
a748ee1a-1d0b-4ae9-9074-279a2b6ba524 2020-10-07 00:00:00 72 1200.6102 760.3020 15.2048 0
a748ee1a-1d0b-4ae9-9074-279a2b6ba524 2020-10-08 00:00:00 70 1100.1120 660.0012 23.7088 0
a748ee1a-1d0b-4ae9-9074-279a2b6ba524 2020-10-09 00:00:00 69 750.3615 734.1516 131.8579 0
FITBIT_HEARTRATE_INTRADAY

RAPIDS_COLUMN_MAPPINGS

RAPIDS column Stream column
TIMESTAMP FLAG_TO_MUTATE
LOCAL_DATE_TIME local_date_time
DEVICE_ID device_id
HEARTRATE heartrate
HEARTRATE_ZONE heartrate_zone

MUTATION

  • COLUMN_MAPPINGS (None)

  • SCRIPTS

    src/data/streams/mutations/fitbit/add_zero_timestamp.py
    

Note

add_zero_timestamp adds an all-zero column called timestamp that will be filled in later in the pipeline by readable_time.R converting LOCAL_DATE_TIME to a unix timestamp taking into account single or multiple time zones.

Example of the raw data RAPIDS expects for this data stream
device_id local_date_time heartrate heartrate_zone
a748ee1a-1d0b-4ae9-9074-279a2b6ba524 2020-10-07 00:00:00 68 outofrange
a748ee1a-1d0b-4ae9-9074-279a2b6ba524 2020-10-07 00:01:00 67 outofrange
a748ee1a-1d0b-4ae9-9074-279a2b6ba524 2020-10-07 00:02:00 67 outofrange
FITBIT_SLEEP_SUMMARY

RAPIDS_COLUMN_MAPPINGS

RAPIDS column Stream column
TIMESTAMP FLAG_TO_MUTATE
LOCAL_DATE_TIME FLAG_TO_MUTATE
LOCAL_START_DATE_TIME local_start_date_time
LOCAL_END_DATE_TIME local_end_date_time
DEVICE_ID device_id
EFFICIENCY efficiency
MINUTES_AFTER_WAKEUP minutes_after_wakeup
MINUTES_ASLEEP minutes_asleep
MINUTES_AWAKE minutes_awake
MINUTES_TO_FALL_ASLEEP minutes_to_fall_asleep
MINUTES_IN_BED minutes_in_bed
IS_MAIN_SLEEP is_main_sleep
TYPE type

MUTATION

  • COLUMN_MAPPINGS (None)

  • SCRIPTS

    - src/data/streams/mutations/fitbit/add_local_date_time.py
    - src/data/streams/mutations/fitbit/add_zero_timestamp.py
    

Note

add_zero_timestamp adds an all-zero column called timestamp that will be filled in later in the pipeline by readable_time.R converting LOCAL_DATE_TIME to a unix timestamp taking into account single or multiple time zones.

Fitbit API has two versions for sleep data, v1 and v1.2. We support both but ignore v1’s count_awake, duration_awake, and count_awakenings, count_restless, duration_restless columns.

Example of the expected raw data
device_id local_start_date_time local_end_date_time efficiency minutes_after_wakeup minutes_asleep minutes_awake minutes_to_fall_asleep minutes_in_bed is_main_sleep type
a748ee1a-1d0b-4ae9-9074-279a2b6ba524 2020-10-10 15:36:30 2020-10-10 16:37:00 92 0 55 5 0 60 0 classic
a748ee1a-1d0b-4ae9-9074-279a2b6ba524 2020-10-10 01:46:30 2020-10-10 08:10:00 88 0 318 65 0 383 1 stages
a748ee1a-1d0b-4ae9-9074-279a2b6ba524 2020-10-11 00:12:30 2020-10-11 11:47:00 89 1 562 132 0 694 1 stages
a748ee1a-1d0b-4ae9-9074-279a2b6ba524 2020-10-12 01:31:00 2020-10-12 09:34:30 93 0 415 68 0 483 1 stages
FITBIT_SLEEP_INTRADAY

RAPIDS_COLUMN_MAPPINGS

RAPIDS column Stream column
TIMESTAMP FLAG_TO_MUTATE
LOCAL_DATE_TIME local_date_time
DEVICE_ID device_id
TYPE_EPISODE_ID type_episode_id
DURATION duration
IS_MAIN_SLEEP is_main_sleep
TYPE type
LEVEL level

MUTATION

  • COLUMN_MAPPINGS (None)

  • SCRIPTS

    src/data/streams/mutations/fitbit/add_zero_timestamp.py
    

Note

add_zero_timestamp adds an all-zero column called timestamp that will be filled in later in the pipeline by readable_time.R converting LOCAL_DATE_TIME to a unix timestamp taking into account single or multiple time zones.

Fitbit API has two versions for sleep data, v1 and v1.2, we support both.

Example of the expected raw data
device_id type_episode_id local_date_time duration level is_main_sleep type
a748ee1a-1d0b-4ae9-9074-279a2b6ba524 0 2020-10-10 15:36:30 60 restless 0 classic
a748ee1a-1d0b-4ae9-9074-279a2b6ba524 0 2020-10-10 15:37:30 660 asleep 0 classic
a748ee1a-1d0b-4ae9-9074-279a2b6ba524 0 2020-10-10 15:48:30 60 restless 0 classic
a748ee1a-1d0b-4ae9-9074-279a2b6ba524
a748ee1a-1d0b-4ae9-9074-279a2b6ba524 1 2020-10-10 01:46:30 420 light 1 stages
a748ee1a-1d0b-4ae9-9074-279a2b6ba524 1 2020-10-10 01:53:30 1230 deep 1 stages
FITBIT_STEPS_SUMMARY

RAPIDS_COLUMN_MAPPINGS

RAPIDS column Stream column
TIMESTAMP FLAG_TO_MUTATE
DEVICE_ID device_id
LOCAL_DATE_TIME local_date_time
STEPS steps

MUTATION

  • COLUMN_MAPPINGS (None)

  • SCRIPTS

    src/data/streams/mutations/fitbit/add_zero_timestamp.py
    

Note

add_zero_timestamp adds an all-zero column called timestamp that will be filled in later in the pipeline by readable_time.R converting LOCAL_DATE_TIME to a unix timestamp taking into account single or multiple time zones.

Example of the expected raw data
device_id local_date_time steps
a748ee1a-1d0b-4ae9-9074-279a2b6ba524 2020-10-07 1775
a748ee1a-1d0b-4ae9-9074-279a2b6ba524 2020-10-08 3201
a748ee1a-1d0b-4ae9-9074-279a2b6ba524 2020-10-09 998
FITBIT_STEPS_INTRADAY

RAPIDS_COLUMN_MAPPINGS

RAPIDS column Stream column
TIMESTAMP FLAG_TO_MUTATE
DEVICE_ID device_id
LOCAL_DATE_TIME local_date_time
STEPS steps

MUTATION

  • COLUMN_MAPPINGS (None)

  • SCRIPTS

    src/data/streams/mutations/fitbit/add_zero_timestamp.py
    

Note

add_zero_timestamp adds an all-zero column called timestamp that will be filled in later in the pipeline by readable_time.R converting LOCAL_DATE_TIME to a unix timestamp taking into account single or multiple time zones.

Example of the expected raw data
device_id local_date_time steps
a748ee1a-1d0b-4ae9-9074-279a2b6ba524 2020-10-07 00:00:00 5
a748ee1a-1d0b-4ae9-9074-279a2b6ba524 2020-10-07 00:01:00 3
a748ee1a-1d0b-4ae9-9074-279a2b6ba524 2020-10-07 00:02:00 0