From 54767f6abbe5e9dfcf4fd35ed3529638572c1ca5 Mon Sep 17 00:00:00 2001 From: kindly Date: Tue, 18 Jun 2024 12:27:36 +0000 Subject: [PATCH] =?UTF-8?q?Deploying=20to=20gh-pages=20from=20@=20kindly/f?= =?UTF-8?q?latterer@91dfbe592e85fd37e368622f8d611e31ee14612d=20?= =?UTF-8?q?=F0=9F=9A=80?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- _sources/changelog.md.txt | 8 + _sources/options.md.txt | 22 ++ changelog.html | 624 ++++++++++++++++++++------------------ objects.inv | Bin 2912 -> 2966 bytes options.html | 168 +++++----- searchindex.js | 2 +- 6 files changed, 450 insertions(+), 374 deletions(-) diff --git a/_sources/changelog.md.txt b/_sources/changelog.md.txt index c1e8bef..0b6b489 100644 --- a/_sources/changelog.md.txt +++ b/_sources/changelog.md.txt @@ -4,6 +4,14 @@ All notable changes to this project will be documented in this file. and this project adheres to [Semantic Versioning](http://semver.org/). +## [0.19.17] - 2024-06-18 + +### New +- truncate postgres + +### Fixed +- timezone date types now accepted in postgres + ## [0.19.15] - 2024-05-09 ### Fixed diff --git a/_sources/options.md.txt b/_sources/options.md.txt index 502011a..267e463 100644 --- a/_sources/options.md.txt +++ b/_sources/options.md.txt @@ -60,6 +60,8 @@ Options: tables to fit data --drop When loading to postgres or sqlite, drop table if already exists. + --truncate When loading to postgres or sqlite, truncate table + if already exists. --id-prefix TEXT Prefix for all `_link` id fields --stats Produce stats about the data in the datapackage.json file @@ -475,6 +477,26 @@ import flatterer flatterer.flatten('inputfile.json', 'ouput_dir', postgres='postgres://user:pass@host/dbname', drop=True) ``` +## Truncate Tables + +**Warning: this could mean you loose data** + +For postgres and sqlite. Truncate the existing table if it exists. This is useful if you want to load the data into a databse with the schema pre-defined. + +### CLI Usage + +```bash +flatterer --postgres='postgres://user:pass@host/dbname' --sqlite-path=sqlite.db INPUT_FILE OUTPUT_DIRECTORY --truncate +``` + +### Python Usage + +```python +import flatterer + +flatterer.flatten('inputfile.json', 'ouput_dir', postgres='postgres://user:pass@host/dbname', truncate=True) +``` + ## Fields File Path to fields CSV file. The fields file can be used for: diff --git a/changelog.html b/changelog.html index 44de662..691ca51 100644 --- a/changelog.html +++ b/changelog.html @@ -202,101 +202,116 @@

Change LogSemantic Versioning.

-

[0.19.15] - 2024-05-09#

+

[0.19.17] - 2024-06-18#

+
+

New#

+
    +

  • truncate postgres

    • +
+

Fixed#

    -

  • nan/inf ignored for xlsx as causing crash

    • +

  • timezone date types now accepted in postgres

-

[0.19.14] - 2024-02-10#

+

[0.19.15] - 2024-05-09#

Fixed#

    -

  • Large xlsx cell values being truncated panic when multi threading.

    • +

  • nan/inf ignored for xlsx as causing crash

-

[0.19.13] - 2024-01-26#

+

[0.19.14] - 2024-02-10#

Fixed#

    -

  • Large xlsx cell values being truncated causing panic if in unicode char.

    • +

  • Large xlsx cell values being truncated panic when multi threading.

-

[0.19.12] - 2023-12-03#

-
-

New#

+

[0.19.13] - 2024-01-26#

+
+

Fixed#

    -

  • Upgrade deps, low_memory option for API

    • +

  • Large xlsx cell values being truncated causing panic if in unicode char.

-
-

[0.19.10] - 2023-10-02#

-

New#

+

[0.19.12] - 2023-12-03#

+
+

New#

    -

  • Upgrade deps, better build times due to latest duckdb

    • +

  • Upgrade deps, low_memory option for API

-
-

[0.19.8] - 2023-06-21#

-

New#

+

[0.19.10] - 2023-10-02#

+
+

New#

    -

  • arrays_as_table option added to convert all arrays to their own table.

    • +

  • Upgrade deps, better build times due to latest duckdb

-
-

[0.19.6] - 2023-06-21#

-

Fixed#

+

[0.19.8] - 2023-06-21#

+
+

New#

    -

  • Errors get raised for postgresql conversion.

    • +

  • arrays_as_table option added to convert all arrays to their own table.

-
-

[0.19.5] - 2023-06-07#

-

Fixed#

+

[0.19.6] - 2023-06-21#

+
+

Fixed#

    -

  • Parquet naming of headers where incorrect for dates.

    • +

  • Errors get raised for postgresql conversion.

-
-

[0.19.4] - 2023-05-26#

-

Fixed#

+

[0.19.5] - 2023-06-07#

+
+

Fixed#

+
    +

  • Parquet naming of headers where incorrect for dates.

    • +
+
+
+
+

[0.19.4] - 2023-05-26#

+
+

Fixed#

  • Allow multiple files while downloading from s3

  • Stop detecting floats where precision is too low.

-
-

[0.19.3] - 2023-05-11#

-
-

Fixed#

+
+

[0.19.3] - 2023-05-11#

+
+

Fixed#

  • CSV output to S3 broken in some cases.

  • Stop csv directory being made when using S3

-
-

[0.19.1] - 2023-05-10#

-
-

New#

+
+

[0.19.1] - 2023-05-10#

+
+

New#

  • JSON Input sources from STDIN, HTTP, S3 and allow all inputs to be GZIPed if have .gz ending.

  • Command line now accepts multiple files from any source.

    • @@ -306,37 +321,37 @@

    New#

-
-

[0.18.0] - 2023-03-17#

-
-

New#

+
+

[0.18.0] - 2023-03-17#

+
+

New#

  • Better type guessing for database inserts.

  • no_link option that removes _link fields in the output.

-
-

[0.17.1] - 2023-01-07#

-
-

Fixed#

+
+

[0.17.1] - 2023-01-07#

+
+

Fixed#

  • Truncate cell that is larger than xlsx allows.

  • Allow more rows in xlsx in non threaded mode.

-
-

[0.17.0] - 2022-12-03#

-
-

New#

+
+

[0.17.0] - 2022-12-03#

+
+

New#

  • Web Assembly version of libflatterer. Available to use here https://lite.flatterer.dev/.

  • Upgrade to vue 3 and vite for web frontend.

-
-

Fixed#

+
+

Fixed#

  • Ignore blank lines in json lines files

  • Better errors when too many files are open

    • @@ -344,46 +359,46 @@

    Fixed#<

-
-

[0.16.2] - 2022-10-25#

-
-

New#

+
+

[0.16.2] - 2022-10-25#

+
+

New#

  • Support python 3.11

-
-

[0.16.2] - 2022-10-16#

-
-

Fixed#

+
+

[0.16.2] - 2022-10-16#

+
+

Fixed#

  • Error not writing larger XLSX files

-
-

[0.16.1] - 2022-10-16#

-
-

Fixed#

+
+

[0.16.1] - 2022-10-16#

+
+

Fixed#

  • Cors for web api

-
-

[0.16.0] - 2022-07-27#

-
-

New#

+
+

[0.16.0] - 2022-07-27#

+
+

New#

  • Local web interface for exploring flatterer features flatterer --web.

-
-

[0.15.0] - 2022-07-27#

-
-

New#

+
+

[0.15.0] - 2022-07-27#

+
+

New#

-
  • Fields File
      +
    • Truncate Tables
      • -
    • Only Fields
        +
      • Fields File
        • -
      • Tables File
          +
        • Only Fields
          • -
        • Only Tables
            +
          • Tables File
            • -
          • Inline One To One
              +
            • Only Tables
              • -
            • Arrays as Table
                +
              • Inline One To One
                • -
              • Schema
                  +
                • Arrays as Table
                  • -
                • Table Prefix
                    +
                  • Schema
                    • -
                  • Path Separator
                      +
                    • Table Prefix
                      • -
                    • Schema Titles
                        +
                      • Path Separator
                        • -
                      • Stats
                          +
                        • Schema Titles
                          • -
                        • Preview
                            +
                          • Stats
                            • -
                          • Threads diff --git a/searchindex.js b/searchindex.js index b92748f..670f599 100644 --- a/searchindex.js +++ b/searchindex.js @@ -1 +1 @@ -Search.setIndex({"docnames": ["changelog", "development", "index", "inputs", "library", "options", "outputs", "s3", "web"], "filenames": ["changelog.md", "development.md", "index.md", "inputs.md", "library.md", "options.md", "outputs.md", "s3.md", "web.md"], "titles": ["Change Log", "Development", "Flatterer. Making JSON flatterer", "Input Sources", "Python Library", "Option Reference", "Outputs", "S3", "Web Interface"], "terms": {"all": [0, 1, 3, 4, 5, 6, 8], "notabl": 0, "thi": [0, 1, 2, 3, 4, 5, 6, 7], "project": 0, "document": 0, "file": [0, 4, 7, 8], "adher": 0, "semant": 0, "version": [0, 5, 7], "nan": 0, "inf": 0, "ignor": 0, "xlsx": [0, 2, 4, 5], "caus": [0, 4, 5], "crash": 0, "larg": [0, 4, 5], "cell": 0, "valu": [0, 5, 8], "being": [0, 5], "truncat": [0, 6], "panic": 0, "when": [0, 2, 5, 6, 7, 8], "multi": 0, "thread": 0, "unicod": 0, "char": 0, "upgrad": 0, "dep": 0, "low_memori": [0, 5], "option": [0, 2, 4, 6, 7, 8], "api": 0, "better": 0, "build": 0, "time": [0, 2, 5, 6], "due": 0, "latest": [0, 7], "duckdb": 0, "arrays_as_t": [0, 5], "convert": [0, 2], "arrai": [0, 2, 4], "own": [0, 2, 5, 6], "tabl": [0, 4], "error": [0, 5], "get": [0, 1, 2, 3, 5, 8], "rais": [0, 5], "postgresql": 0, "convers": [0, 4], "parquet": [0, 2, 5, 7], "name": [0, 2, 4, 6], "header": [0, 5, 6], "where": [0, 2, 4, 5, 6], "incorrect": 0, "date": [0, 2, 4, 5, 6], "allow": [0, 5, 6], "multipl": [0, 5], "while": 0, "download": [0, 5], "from": [0, 2, 3, 4, 5, 6, 7], "s3": [0, 2], "stop": [0, 5], "detect": 0, "float": 0, "precis": 0, "i": [0, 2, 3, 4, 5, 6, 7], "too": [0, 2], "low": 0, "csv": [0, 4, 5, 7], "output": [0, 1, 3, 7], "broken": 0, "some": [0, 2, 5, 6], "case": [0, 4, 5], "directori": [0, 3, 4, 5, 6], "made": [0, 6], "us": [0, 2, 3, 4, 5, 6, 7], "json": [0, 3, 4, 7], "input": [0, 2, 4, 5, 6, 7], "sourc": [0, 1, 2, 4], "stdin": [0, 7], "http": [0, 5, 7], "gzipe": [0, 3], "have": [0, 2, 5, 6], "gz": [0, 3], "end": [0, 2, 3, 5], "command": [0, 2, 3, 4, 5, 6, 8], "line": [0, 2, 3, 4, 6, 8], "now": [0, 4], "accept": [0, 7, 8], "ani": [0, 5, 8], "stream": [0, 2, 4, 7], "path": [0, 3, 4, 6, 7], "queri": [0, 5], "filter": 0, "object": [0, 2, 4, 5], "stat": 0, "produc": [0, 5, 6], "statist": [0, 5], "ouput": 0, "type": [0, 2, 5, 6], "guess": [0, 2, 6], "databas": [0, 2, 4, 5, 6], "insert": [0, 2, 5], "no_link": [0, 5], "_link": [0, 2, 4, 5, 6], "field": [0, 4, 7], "larger": 0, "than": [0, 2, 5], "more": [0, 2, 4, 5], "row": [0, 5], "non": [0, 4, 5], "mode": [0, 5, 6], "web": [0, 3, 5], "assembli": 0, "libflatter": 0, "avail": 0, "here": 0, "lite": 0, "flatter": [0, 1, 3, 4, 5, 6, 7, 8], "dev": 0, "vue": 0, "vite": 0, "frontend": 0, "blank": 0, "mani": [0, 5, 6], "ar": [0, 2, 3, 4, 5, 6, 7], "open": [0, 4, 8], "could": [0, 2, 4, 5], "support": [0, 2, 7], "python": [0, 1, 3], "write": 0, "cor": 0, "local": [0, 5, 7], "interfac": 0, "explor": [0, 2, 8], "featur": [0, 2, 8], "evolv": 0, "sqlite": [0, 2, 4, 5], "postgr": [0, 2], "can": [0, 2, 3, 4, 5, 6, 7, 8], "add": [0, 4, 5], "data": [0, 2, 4, 5, 6, 7, 8], "exist": [0, 5], "alter": [0, 5], "need": [0, 1, 2, 4, 5, 6, 7, 8], "drop": 0, "No": 0, "Will": [0, 5], "temp": 0, "space": [0, 4, 5], "suppli": [0, 4, 5], "switch": 0, "connect": [0, 3, 5, 7], "environ": [0, 1, 5, 7, 8], "variabl": [0, 1, 5, 7, 8], "sql_script": [0, 5, 6], "export": [0, 5, 7], "script": [0, 6], "make": [0, 4, 5], "backward": 0, "compat": 0, "earlier": 0, "pushdown": 0, "copi": [0, 5, 6], "top": [0, 5], "level": [0, 5], "down": [0, 5], "child": [0, 5], "one": [0, 2, 5, 6], "ha": [0, 2, 5], "its": [0, 2, 5, 6], "kei": [0, 2, 4, 5], "id": [0, 2, 4, 5, 6], "you": [0, 2, 3, 4, 5], "want": [0, 2, 4, 5], "relat": [0, 2, 5, 6], "also": [0, 2, 3, 4, 5, 7], "denorm": 0, "so": [0, 2, 4, 5, 8], "common": [0, 5], "requir": [0, 2, 5, 6, 7], "less": [0, 5], "join": [0, 2, 5], "string": [0, 4, 5], "schema": [0, 6, 7], "choos": [0, 5], "If": [0, 2, 3, 4, 5], "alreadi": [0, 5], "longer": 0, "actual": 0, "creat": [0, 1, 5, 6], "onc": [0, 1, 2, 7], "run": [0, 1, 4, 5, 6, 8], "core": [0, 5], "work": [0, 2, 3, 4, 5], "best": [0, 2, 5], "ndjson": [0, 3], "break": 0, "renam": [0, 2], "same": [0, 2, 3, 4, 5, 6], "wai": [0, 5, 7, 8], "old": [0, 5], "concaton": [0, 5], "datapackag": [0, 2, 5], "correct": [0, 5], "list": [0, 2, 4, 5], "escap": [0, 5], "quot": 0, "clearer": 0, "happen": 0, "rust": [0, 1, 2], "catch": 0, "certain": [0, 5], "mai": [0, 2, 5], "forign": 0, "decim": 0, "lower": [0, 5], "memori": [0, 2, 4], "index": 0, "foreign": [0, 2], "contraint": 0, "main_table_nam": [0, 5], "wa": [0, 2, 5], "number": [0, 2, 4, 5, 6], "except": [0, 2], "flatten": [0, 2, 4, 5, 7, 8], "correctli": 0, "function": [0, 7], "iter": [0, 4], "doc": [0, 7], "test": [0, 1], "iterator_flatten": 0, "deprec": 0, "just": [0, 5], "subset": 0, "lenient": 0, "tmp": 0, "delet": [0, 5], "preview": 0, "cli": 0, "librari": [0, 3], "save": [0, 5], "main": [0, 4], "yaglish": 0, "parser": [0, 2], "both": [0, 5], "schema_guess": 0, "tell": 0, "an": [0, 2, 4, 5], "empti": [0, 5], "do": [0, 2, 4, 5], "ctrlc": 0, "improv": [0, 5], "traceback": 0, "shown": 0, "occur": 0, "folder": [0, 5], "order": [0, 2, 5], "control": 0, "tab": [0, 5], "begin": 0, "handl": 0, "long": [0, 2, 3], "excel": [0, 5], "sheet": [0, 2, 5, 6], "see": [0, 2, 3, 4, 5], "github": [0, 1], "com": [0, 3, 5, 7], "kindli": 0, "issu": [0, 4, 5], "field_typ": [0, 2, 4, 5, 6], "redund": 0, "_link_": 0, "main_tabl": 0, "human": 0, "readabl": 0, "messag": [0, 5], "bad": 0, "charact": [0, 5], "strip": 0, "warn": [0, 4, 5], "check": [0, 5], "limit": [0, 6], "found": [0, 6], "lot": [0, 2, 5], "edg": 0, "instead": [0, 5], "leav": 0, "unus": 0, "partial": 0, "unwrap": 0, "channel": 0, "send": 0, "possibl": [0, 4], "reflect": 0, "lib": 0, "flatfil": 0, "new_with_dafualt": 0, "verbos": 0, "insta": 0, "mean": [0, 2, 5, 7], "onli": [0, 2, 4, 7], "show": [0, 4, 5], "specifi": [0, 2, 5, 7], "start": [0, 5], "root": [0, 5], "clippi": 0, "lint": 0, "metadata": [0, 2, 4, 6], "minor": 0, "speedup": 0, "format": [0, 2, 3, 7], "much": [0, 2], "abil": 0, "field_titl": [0, 2, 4, 5], "column": [0, 2, 5], "head": [0, 2, 5, 6], "titl": [0, 2, 4, 6], "out": [0, 2, 5], "jsonschema": [0, 5], "speed": [0, 5], "smartstr": 0, "smallvec": 0, "pypi": 0, "action": 0, "changelog": 0, "regress": 0, "anyhow": 0, "give": [0, 5], "context": 0, "prefix": 0, "namespec": 0, "separ": [0, 2], "_": [0, 2, 5], "them": 0, "load": [0, 2, 5, 6], "wheel": [0, 1, 2], "window": [0, 2, 8], "maco": [0, 2], "automat": [0, 8], "publish": 0, "inlin": 0, "One": 0, "item": [0, 4, 5], "treat": [0, 3, 5], "sub": [0, 2, 5], "pleas": 1, "contribut": 1, "stabl": 1, "toolchain": [1, 2], "virtualenv": 1, "clone": 1, "repo": 1, "virtual": 1, "ve": 1, "bin": 1, "activ": 1, "maturin": 1, "tool": [1, 2, 3, 4, 6], "extens": [1, 2, 5, 6], "pip": [1, 2], "releas": 1, "anylinux": [1, 2], "3": [1, 2, 4, 5, 6], "6": [1, 2, 4, 5], "distribut": 1, "sudo": 1, "docker": 1, "t": [1, 5], "To": [1, 8], "v": 1, "pwd": 1, "io": 1, "set": [1, 4, 5, 7, 8], "enviro": [1, 5], "flatterer_log": 1, "trace": 1, "full": [1, 3, 5], "debug": 1, "opinion": 2, "which": [2, 4, 5, 6, 7], "tri": 2, "analysi": [2, 7], "receiv": 2, "structur": [2, 5, 6], "deepli": 2, "nest": 2, "well": 2, "hard": 2, "determin": [2, 5], "what": [2, 5], "contain": [2, 4, 5, 6], "even": [2, 5], "after": [2, 5], "know": 2, "how": [2, 3], "part": 2, "pipelin": 2, "aim": 2, "first": [2, 4, 5], "go": [2, 4, 5, 6], "face": 2, "abov": [2, 3, 4, 5], "problem": 2, "It": [2, 4, 5, 6], "up": [2, 5], "your": [2, 4, 5], "hand": 2, "written": [2, 5], "benefit": 2, "over": [2, 5, 6], "most": [2, 5, 7], "approach": 2, "fast": 2, "bind": 2, "eas": 2, "10x": 2, "faster": 2, "effici": 2, "custom": 2, "practic": 2, "been": 2, "learnt": 2, "countless": 2, "gener": [2, 4, 5], "parent": [2, 5], "greater": 2, "binari": 2, "linux": 2, "x64": 2, "univers": 2, "x86": 2, "On": 2, "other": [2, 4, 5], "platform": [2, 4, 5, 6], "sai": [2, 5], "like": [2, 5, 6, 7], "game": [2, 3, 4, 5, 6], "1": [2, 4, 5, 6, 8], "A": [2, 4, 5, 6, 8], "released": [2, 4, 5, 6], "2015": [2, 4, 5], "01": [2, 4, 5], "xbox": 2, "playstat": 2, "rate": 2, "code": 2, "e": [2, 4, 5, 7], "everyon": [2, 4], "2": [2, 4, 5, 6, 7], "b": [2, 4, 5], "2016": [2, 4, 5], "pc": [2, 5], "games_dir": [2, 3, 4, 6], "refer": [2, 4], "detail": 2, "addit": [2, 5], "inform": [2, 4, 7], "configur": [2, 3, 8], "import": [2, 4, 5, 6], "By": [2, 5], "follow": [2, 5, 6, 7, 8], "tree": 2, "rating_cod": [2, 4, 6], "rating_nam": [2, 4, 6], "0": [2, 4, 5, 8], "special": 2, "primari": 2, "uniqu": [2, 5], "per": [2, 6], "promot": 2, "relationship": [2, 5, 6], "properti": [2, 5], "therefor": [2, 5], "_link_gam": [2, 5, 6], "everi": [2, 5], "easili": [2, 6], "would": [2, 5], "_link_platform": [2, 5], "_link__": 2, "table_nam": [2, 4, 5, 6], "referenc": 2, "about": [2, 4, 5, 6], "field_nam": [2, 4, 5, 6], "count": [2, 4, 5, 6], "text": [2, 4, 5, 6], "The": [2, 3, 4, 5, 6, 7], "initi": 2, "edit": [2, 5], "rerun": 2, "transform": 2, "new_games_dir": 2, "f": [2, 4, 5, 6], "myfield": [2, 5], "rearrang": [2, 5], "remov": [2, 5, 6], "flag": 2, "includ": [2, 5], "tabular": [2, 6], "datapackg": [2, 6], "spec": [2, 6], "retriev": 3, "varieti": 3, "below": 3, "exampl": [3, 4, 5], "thei": [3, 5], "rel": 3, "cat": [3, 6], "my": [3, 5], "bucketnam": 3, "bucket": [3, 7], "assum": [3, 5], "compress": 3, "select": [3, 5], "last": 3, "argument": [3, 4], "rest": 3, "games2": [3, 4], "games3": 3, "As": [3, 4, 5], "each": [3, 5, 6], "variou": 3, "For": [3, 4, 5], "new": [3, 6], "delimit": [3, 5], "repres": [4, 5, 6], "howev": [4, 5, 7], "dict": 4, "list_of_object": 4, "complic": [4, 5], "feed": 4, "ijson": 4, "def": 4, "array_item_gener": 4, "fixtur": 4, "basic": [4, 6], "rb": 4, "manipul": 4, "modifi": [4, 5], "yield": 4, "befor": [4, 5], "process": [4, 5], "byte": 4, "interpret": 4, "true": [4, 5, 6, 7], "describ": 4, "print": 4, "fiel": 4, "5": 4, "table_titl": [4, 5, 6], "develop": 4, "db": [4, 5, 6], "4": [4, 5], "7": 4, "_link_main": 4, "8": 4, "9": 4, "10": [4, 5], "11": 4, "similar": 4, "locat": 4, "fals": [4, 5], "locatin": 4, "dataset": [4, 5], "put": [4, 5, 6], "rating_co": 4, "system": [4, 5], "temporari": [4, 7], "g": [4, 5, 7], "look": [5, 6], "output_directori": 5, "base": 5, "nocsv": 5, "default": [5, 8], "noxlsx": 5, "nosqlit": 5, "noparquet": 5, "d": 5, "n": 5, "m": 5, "without": [5, 6, 7, 8], "p": 5, "j": 5, "o": 5, "l": 5, "singl": 5, "y": 5, "alwai": [5, 7], "": 5, "url": 5, "seper": 5, "denot": 5, "within": [5, 6], "h": 5, "given": 5, "slug": 5, "underscore_slug": 5, "w": 5, "integ": 5, "amount": [5, 6], "final": 5, "result": 5, "cpu": 5, "fit": 5, "exit": 5, "input_fil": 5, "user": [5, 6], "pass": 5, "host": [5, 6, 8], "dbname": 5, "should": 5, "In": 5, "env": 5, "database_url": 5, "my_env_var": 5, "envir": 5, "inputfil": 5, "ouput_dir": 5, "defin": [5, 7], "extra": 5, "appear": [5, 6], "ram": 5, "4gb": 5, "game_id": 5, "platforms_requir": 5, "platforms_nam": 5, "platforms_id": 5, "push": 5, "trust": 5, "suffici": 5, "back": 5, "togeth": 5, "analys": 5, "delimet": 5, "fastest": 5, "pars": 5, "jl": 5, "sometim": 5, "call": 5, "doe": [5, 6], "json_stream": 5, "goessner": 5, "net": 5, "articl": 5, "jsonpath": 5, "express": 5, "particular": 5, "evalu": 5, "against": 5, "null": 5, "film": 5, "second": 5, "care": 5, "json_path": 5, "logic": 5, "condit": 5, "either": 5, "OR": 5, "myschema": 5, "postgres_schema": 5, "differ": 5, "come": 5, "rule": 5, "conflict": 5, "coerc": 5, "origin": 5, "kept": 5, "still": 5, "id_prefix": 5, "random": 5, "ad": 5, "across": 5, "recommend": 5, "conatain": 5, "Not": 5, "complet": 5, "parallel": 5, "safe": 5, "two": 5, "try": 5, "currupt": 5, "retri": 5, "loos": 5, "chang": 5, "unwant": 5, "miss": 5, "match": 5, "eventu": 5, "easiest": 5, "sure": 5, "behaviour": 5, "exclud": 5, "only_field": 5, "tables_nam": 5, "basi": 5, "tabless": 5, "only_t": 5, "ever": 5, "associ": 5, "With": 5, "under": 5, "inline_one_to_on": 5, "encod": 5, "remot": 5, "server": [5, 6], "otherwis": 5, "mytabl": 5, "myprefix_": 5, "myprefix_myt": 5, "namespac": 5, "table_prefix": 5, "myobject_myfield": 5, "myobject": 5, "theres": 5, "my_object": 5, "my_field": 5, "my_object_my_field": 5, "confus": 5, "might": 5, "expect": 5, "clash": 5, "fix": 5, "whatev": 5, "___": 5, "my_object__my_field": 5, "__": 5, "path_separo": 5, "take": 5, "modif": 5, "slugifi": 5, "charactor": 5, "alphanumer": 5, "replac": 5, "strang": 5, "turn": 5, "previou": 5, "my_strange_titl": 5, "left": 5, "schema_titl": 5, "underscore_slug_": 5, "done": 5, "x3": 5, "x1": 5, "24": 5, "worth": 5, "lead": [5, 7], "perform": 5, "veri": 5, "100": 5, "slower": 5, "ensur": 5, "circumst": 5, "wrong": 5, "chosen": 5, "reduc": 5, "sacrif": 5, "int": 6, "unless": [6, 7], "help": 6, "definit": 6, "numer": 6, "timestamp": 6, "psql": 6, "someth": 6, "password": 6, "adapt": 6, "plain": 6, "advantag": 6, "games_dir_": 6, "sqlite3": 6, "my_databas": 6, "content": 6, "skip": 6, "profil": 6, "packag": 6, "resourc": 6, "primarykei": 6, "fetch": 7, "provid": 7, "aws_default_region": 7, "u": 7, "east": 7, "aws_access_key_id": 7, "access_key_id": 7, "aws_secret_access_kei": 7, "secret_access_kei": 7, "aws_endpoint": 7, "endpoint": 7, "aws_session_token": 7, "token": 7, "aws_container_credentials_relative_uri": 7, "aw": 7, "amazon": 7, "amazonec": 7, "developerguid": 7, "task": 7, "iam": 7, "role": 7, "html": 7, "aws_allow_http": 7, "permit": 7, "tl": 7, "commonli": 7, "alon": 7, "upload": 7, "someaccesskeyid": 7, "somesecretaccesskeyid": 7, "mybucket": 7, "mydata": 7, "flattenedoutput": 7, "store": 7, "constrain": 7, "serverless": 7, "read": 7, "twice": 7, "known": 7, "upfront": 7, "learn": 8, "visual": 8, "feedback": 8, "webserv": 8, "port": 8, "8080": 8, "localhost": 8, "127": 8, "max_siz": 8, "size": 8, "mb": 8, "maximum": 8, "500": 8, "no_brows": 8, "browser": 8}, "objects": {}, "objtypes": {}, "objnames": {}, "titleterms": {"chang": 0, "log": [0, 1], "0": 0, "19": 0, "15": 0, "2024": 0, "05": 0, "09": 0, "fix": 0, "14": 0, "02": 0, "10": 0, "13": 0, "01": 0, "26": 0, "12": 0, "2023": 0, "03": 0, "new": [0, 5], "8": 0, "06": 0, "21": 0, "6": 0, "5": 0, "07": 0, "4": 0, "3": 0, "11": 0, "1": 0, "18": 0, "17": 0, "2022": 0, "16": 0, "2": 0, "25": 0, "27": 0, "23": 0, "22": 0, "remov": 0, "24": 0, "04": 0, "28": 0, "9": 0, "7": 0, "2021": 0, "ad": 0, "20": 0, "develop": 1, "local": [1, 2, 3], "instal": [1, 2], "build": 1, "flatter": 2, "make": 2, "json": [2, 5, 6], "introduct": 2, "rational": 2, "exampl": [2, 7], "run": 2, "cli": [2, 5], "web": [2, 8], "interfac": [2, 8], "As": 2, "python": [2, 4, 5], "librari": [2, 4], "output": [2, 4, 5, 6], "file": [2, 3, 5, 6], "main": [2, 5], "tabl": [2, 5, 6], "One": [2, 5], "To": [2, 5], "mani": 2, "link": [2, 5], "field": [2, 5, 6], "csv": [2, 6], "input": 3, "sourc": 3, "stdin": 3, "http": 3, "": 3, "s3": [3, 7], "gzip": 3, "multipl": 3, "usag": [4, 5, 7], "creat": 4, "panda": 4, "datafram": 4, "option": 5, "refer": 5, "help": 5, "small": 5, "onli": 5, "format": 5, "name": 5, "pushdown": 5, "No": 5, "path": 5, "arrai": 5, "line": 5, "delemit": 5, "ndjson": 5, "stream": 5, "filter": 5, "forc": 5, "postgr": [5, 6], "schema": 5, "evolv": 5, "drop": 5, "inlin": 5, "prefix": 5, "separ": 5, "titl": 5, "stat": 5, "preview": 5, "thread": 5, "sql": [5, 6], "script": 5, "low": 5, "memori": 5, "api": 5, "xlsx": 6, "sqlite": 6, "parquet": 6, "postgresql": 6, "postgresql_schema": 6, "postgresql_load": 6, "sqlite_schema": 6, "sqlite_load": 6, "datapackag": 6, "url": 7, "region": 7, "authent": 7, "note": 7}, "envversion": {"sphinx.domains.c": 2, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 8, "sphinx.domains.index": 1, "sphinx.domains.javascript": 2, "sphinx.domains.math": 2, "sphinx.domains.python": 3, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx": 57}, "alltitles": {"Change Log": [[0, "change-log"]], "[0.19.15] - 2024-05-09": [[0, "id1"]], "Fixed": [[0, "fixed"], [0, "id3"], [0, "id5"], [0, "id12"], [0, "id14"], [0, "id16"], [0, "id18"], [0, "id24"], [0, "id27"], [0, "id31"], [0, "id33"], [0, "id41"], [0, "id55"], [0, "id57"], [0, "id61"], [0, "id63"], [0, "id67"], [0, "id69"], [0, "id74"], [0, "id78"], [0, "id82"], [0, "id84"], [0, "id93"], [0, "id95"]], "[0.19.14] - 2024-02-10": [[0, "id2"]], "[0.19.13] - 2024-01-26": [[0, "id4"]], "[0.19.12] - 2023-12-03": [[0, "id6"]], "New": [[0, "new"], [0, "id8"], [0, "id10"], [0, "id20"], [0, "id22"], [0, "id26"], [0, "id29"], [0, "id35"], [0, "id37"], [0, "id39"], [0, "id43"], [0, "id45"], [0, "id48"], [0, "id71"], [0, "id73"], [0, "id76"], [0, "id80"], [0, "id86"]], "[0.19.10] - 2023-10-02": [[0, "id7"]], "[0.19.8] - 2023-06-21": [[0, "id9"]], "[0.19.6] - 2023-06-21": [[0, "id11"]], "[0.19.5] - 2023-06-07": [[0, "id13"]], "[0.19.4] - 2023-05-26": [[0, "id15"]], "[0.19.3] - 2023-05-11": [[0, "id17"]], "[0.19.1] - 2023-05-10": [[0, "id19"]], "[0.18.0] - 2023-03-17": [[0, "id21"]], "[0.17.1] - 2023-01-07": [[0, "id23"]], "[0.17.0] - 2022-12-03": [[0, "id25"]], "[0.16.2] - 2022-10-25": [[0, "id28"]], "[0.16.2] - 2022-10-16": [[0, "id30"]], "[0.16.1] - 2022-10-16": [[0, "id32"]], "[0.16.0] - 2022-07-27": [[0, "id34"]], "[0.15.0] - 2022-07-27": [[0, "id36"]], "Changed": [[0, "changed"], [0, "id49"], [0, "id51"], [0, "id53"], [0, "id59"], [0, "id65"], [0, "id77"], [0, "id81"], [0, "id85"], [0, "id88"], [0, "id90"], [0, "id97"]], "[0.14.2] - 2022-07-27": [[0, "id38"]], "[0.14.1] - 2022-07-23": [[0, "id40"]], "[0.14.0] - 2022-07-22": [[0, "id42"]], "Removed": [[0, "removed"]], "[0.13.2] - 2022-06-24": [[0, "id44"]], "[0.13.1] - 2022-05-27": [[0, "id46"]], "[0.13.0] - 2022-04-28": [[0, "id47"]], "[0.12.12] - 2022-04-15": [[0, "id50"]], "[0.12.11] - 2022-04-09": [[0, "id52"]], "[0.12.10] - 2022-03-21": [[0, "id54"]], "[0.12.9] - 2022-03-07": [[0, "id56"]], "[0.12.8] - 2022-03-01": [[0, "id58"]], "[0.12.7] - 2022-02-27": [[0, "id60"]], "[0.12.6] - 2022-02-26": [[0, "id62"]], "[0.12.4] - 2022-02-18": [[0, "id64"]], "[0.12.3] - 2022-02-15": [[0, "id66"]], "[0.12.1] - 2022-02-15": [[0, "id68"]], "[0.12] - 2022-02-02": [[0, "id70"]], "[0.11] - 2022-01-25": [[0, "id72"]], "[0.10.1] - 2021-12-22": [[0, "id75"]], "[0.10] - 2021-12-22": [[0, "id79"]], "[0.9] - 2021-12-12": [[0, "id83"]], "[0.8] - 2021-12-01": [[0, "id87"]], "[0.7.1] - 2021-12-01": [[0, "id89"]], "[0.7] - 2021-11-24": [[0, "id91"]], "Added": [[0, "added"], [0, "id99"], [0, "id101"], [0, "id103"]], "[0.6.2] - 2021-11-24": [[0, "id92"]], "[0.6.1] - 2021-11-24": [[0, "id94"]], "[0.6] - 2021-11-23": [[0, "id96"]], "[0.5] - 2021-11-22": [[0, "id98"]], "[0.4] - 2021-11-20": [[0, "id100"]], "[0.3] - 2021-10-25": [[0, "id102"]], "Development": [[1, "development"]], "Local install": [[1, "local-install"]], "Builds": [[1, "builds"]], "Logging": [[1, "logging"]], "Flatterer. Making JSON flatterer": [[2, "flatterer-making-json-flatterer"]], "Introduction": [[2, "introduction"]], "Rationale": [[2, "rationale"]], "Install": [[2, "install"]], "Example JSON": [[2, "example-json"]], "Running Flatterer": [[2, "running-flatterer"]], "CLI": [[2, "cli"]], "Local web interface": [[2, "local-web-interface"]], "As python library": [[2, "as-python-library"]], "Output Files": [[2, "output-files"]], "Main Table": [[2, "main-table"]], "One To Many Table": [[2, "one-to-many-table"]], "Link Fields": [[2, "link-fields"]], "Fields CSV": [[2, "fields-csv"]], "Input Sources": [[3, "input-sources"]], "Local File": [[3, "local-file"]], "Stdin": [[3, "stdin"]], "HTTP(s)": [[3, "http-s"]], "S3": [[3, "s3"], [7, "s3"]], "GZIP": [[3, "gzip"]], "Multiple inputs": [[3, "multiple-inputs"]], "Python Library": [[4, "python-library"]], "Usage": [[4, "usage"]], "Output": [[4, "output"]], "Creating pandas DataFrames": [[4, "creating-pandas-dataframes"]], "Option Reference": [[5, "option-reference"]], "Help (CLI Only) ": [[5, "help-small-cli-only-small"]], "Output Formats": [[5, "output-formats"]], "CLI Usage": [[5, "cli-usage"], [5, "id1"], [5, "id3"], [5, "id5"], [5, "id7"], [5, "id9"], [5, "id11"], [5, "id13"], [5, "id15"], [5, "id17"], [5, "id19"], [5, "id21"], [5, "id23"], [5, "id25"], [5, "id27"], [5, "id29"], [5, "id31"], [5, "id33"], [5, "id35"], [5, "id37"], [5, "id39"], [5, "id41"], [5, "id43"], [5, "id45"], [5, "id47"]], "Python Usage": [[5, "python-usage"], [5, "id2"], [5, "id4"], [5, "id6"], [5, "id8"], [5, "id10"], [5, "id12"], [5, "id14"], [5, "id16"], [5, "id18"], [5, "id20"], [5, "id22"], [5, "id24"], [5, "id26"], [5, "id28"], [5, "id30"], [5, "id32"], [5, "id34"], [5, "id36"], [5, "id38"], [5, "id40"], [5, "id42"], [5, "id44"], [5, "id46"], [5, "id48"], [5, "id49"], [5, "id50"]], "Main Table Name": [[5, "main-table-name"]], "Pushdown Fields": [[5, "pushdown-fields"]], "No Link Fields": [[5, "no-link-fields"]], "Path to JSON Array": [[5, "path-to-json-array"]], "New Line Delemited JSON (NDJSON)": [[5, "new-line-delemited-json-ndjson"]], "JSON Stream": [[5, "json-stream"]], "JSON Path Filter": [[5, "json-path-filter"]], "Force": [[5, "force"]], "Postgres Schema": [[5, "postgres-schema"]], "Evolve Tables": [[5, "evolve-tables"]], "Drop Tables": [[5, "drop-tables"]], "Fields File": [[5, "fields-file"]], "Only Fields": [[5, "only-fields"]], "Tables File": [[5, "tables-file"]], "Only Tables": [[5, "only-tables"]], "Inline One To One": [[5, "inline-one-to-one"]], "Arrays as Table": [[5, "arrays-as-table"]], "Schema": [[5, "schema"]], "Table Prefix": [[5, "table-prefix"]], "Path Separator": [[5, "path-separator"]], "Schema Titles": [[5, "schema-titles"]], "Stats": [[5, "stats"]], "Preview": [[5, "preview"]], "Threads": [[5, "threads"]], "Sql Scripts": [[5, "sql-scripts"]], "Low Memory (api only)": [[5, "low-memory-api-only"]], "Outputs": [[6, "outputs"]], "CSV": [[6, "csv"]], "XLSX": [[6, "xlsx"]], "SQLITE": [[6, "sqlite"]], "POSTGRES": [[6, "postgres"]], "PARQUET": [[6, "parquet"]], "fields.csv": [[6, "fields-csv"]], "tables.csv": [[6, "tables-csv"]], "Postgresql Files": [[6, "postgresql-files"]], "postgresql_schema.sql": [[6, "postgresql-schema-sql"]], "postgresql_load.sql": [[6, "postgresql-load-sql"]], "Sqlite Files.": [[6, "sqlite-files"]], "sqlite_schema.sql": [[6, "sqlite-schema-sql"]], "sqlite_load.sql": [[6, "sqlite-load-sql"]], "datapackage.json": [[6, "datapackage-json"]], "URL": [[7, "url"]], "Region and authentication.": [[7, "region-and-authentication"]], "Example": [[7, "example"]], "Usage Note": [[7, "usage-note"]], "Web Interface": [[8, "web-interface"]]}, "indexentries": {}}) \ No newline at end of file +Search.setIndex({"docnames": ["changelog", "development", "index", "inputs", "library", "options", "outputs", "s3", "web"], "filenames": ["changelog.md", "development.md", "index.md", "inputs.md", "library.md", "options.md", "outputs.md", "s3.md", "web.md"], "titles": ["Change Log", "Development", "Flatterer. Making JSON flatterer", "Input Sources", "Python Library", "Option Reference", "Outputs", "S3", "Web Interface"], "terms": {"all": [0, 1, 3, 4, 5, 6, 8], "notabl": 0, "thi": [0, 1, 2, 3, 4, 5, 6, 7], "project": 0, "document": 0, "file": [0, 4, 7, 8], "adher": 0, "semant": 0, "version": [0, 5, 7], "truncat": [0, 6], "postgr": [0, 2], "timezon": 0, "date": [0, 2, 4, 5, 6], "type": [0, 2, 5, 6], "now": [0, 4], "accept": [0, 7, 8], "nan": 0, "inf": 0, "ignor": 0, "xlsx": [0, 2, 4, 5], "caus": [0, 4, 5], "crash": 0, "larg": [0, 4, 5], "cell": 0, "valu": [0, 5, 8], "being": [0, 5], "panic": 0, "when": [0, 2, 5, 6, 7, 8], "multi": 0, "thread": 0, "unicod": 0, "char": 0, "upgrad": 0, "dep": 0, "low_memori": [0, 5], "option": [0, 2, 4, 6, 7, 8], "api": 0, "better": 0, "build": 0, "time": [0, 2, 5, 6], "due": 0, "latest": [0, 7], "duckdb": 0, "arrays_as_t": [0, 5], "convert": [0, 2], "arrai": [0, 2, 4], "own": [0, 2, 5, 6], "tabl": [0, 4], "error": [0, 5], "get": [0, 1, 2, 3, 5, 8], "rais": [0, 5], "postgresql": 0, "convers": [0, 4], "parquet": [0, 2, 5, 7], "name": [0, 2, 4, 6], "header": [0, 5, 6], "where": [0, 2, 4, 5, 6], "incorrect": 0, "allow": [0, 5, 6], "multipl": [0, 5], "while": 0, "download": [0, 5], "from": [0, 2, 3, 4, 5, 6, 7], "s3": [0, 2], "stop": [0, 5], "detect": 0, "float": 0, "precis": 0, "i": [0, 2, 3, 4, 5, 6, 7], "too": [0, 2], "low": 0, "csv": [0, 4, 5, 7], "output": [0, 1, 3, 7], "broken": 0, "some": [0, 2, 5, 6], "case": [0, 4, 5], "directori": [0, 3, 4, 5, 6], "made": [0, 6], "us": [0, 2, 3, 4, 5, 6, 7], "json": [0, 3, 4, 7], "input": [0, 2, 4, 5, 6, 7], "sourc": [0, 1, 2, 4], "stdin": [0, 7], "http": [0, 5, 7], "gzipe": [0, 3], "have": [0, 2, 5, 6], "gz": [0, 3], "end": [0, 2, 3, 5], "command": [0, 2, 3, 4, 5, 6, 8], "line": [0, 2, 3, 4, 6, 8], "ani": [0, 5, 8], "stream": [0, 2, 4, 7], "path": [0, 3, 4, 6, 7], "queri": [0, 5], "filter": 0, "object": [0, 2, 4, 5], "stat": 0, "produc": [0, 5, 6], "statist": [0, 5], "ouput": 0, "guess": [0, 2, 6], "databas": [0, 2, 4, 5, 6], "insert": [0, 2, 5], "no_link": [0, 5], "_link": [0, 2, 4, 5, 6], "field": [0, 4, 7], "larger": 0, "than": [0, 2, 5], "more": [0, 2, 4, 5], "row": [0, 5], "non": [0, 4, 5], "mode": [0, 5, 6], "web": [0, 3, 5], "assembli": 0, "libflatter": 0, "avail": 0, "here": 0, "lite": 0, "flatter": [0, 1, 3, 4, 5, 6, 7, 8], "dev": 0, "vue": 0, "vite": 0, "frontend": 0, "blank": 0, "mani": [0, 5, 6], "ar": [0, 2, 3, 4, 5, 6, 7], "open": [0, 4, 8], "could": [0, 2, 4, 5], "support": [0, 2, 7], "python": [0, 1, 3], "write": 0, "cor": 0, "local": [0, 5, 7], "interfac": 0, "explor": [0, 2, 8], "featur": [0, 2, 8], "evolv": 0, "sqlite": [0, 2, 4, 5], "can": [0, 2, 3, 4, 5, 6, 7, 8], "add": [0, 4, 5], "data": [0, 2, 4, 5, 6, 7, 8], "exist": [0, 5], "alter": [0, 5], "need": [0, 1, 2, 4, 5, 6, 7, 8], "drop": 0, "No": 0, "Will": [0, 5], "temp": 0, "space": [0, 4, 5], "suppli": [0, 4, 5], "switch": 0, "connect": [0, 3, 5, 7], "environ": [0, 1, 5, 7, 8], "variabl": [0, 1, 5, 7, 8], "sql_script": [0, 5, 6], "export": [0, 5, 7], "script": [0, 6], "make": [0, 4, 5], "backward": 0, "compat": 0, "earlier": 0, "pushdown": 0, "copi": [0, 5, 6], "top": [0, 5], "level": [0, 5], "down": [0, 5], "child": [0, 5], "one": [0, 2, 5, 6], "ha": [0, 2, 5], "its": [0, 2, 5, 6], "kei": [0, 2, 4, 5], "id": [0, 2, 4, 5, 6], "you": [0, 2, 3, 4, 5], "want": [0, 2, 4, 5], "relat": [0, 2, 5, 6], "also": [0, 2, 3, 4, 5, 7], "denorm": 0, "so": [0, 2, 4, 5, 8], "common": [0, 5], "requir": [0, 2, 5, 6, 7], "less": [0, 5], "join": [0, 2, 5], "string": [0, 4, 5], "schema": [0, 6, 7], "choos": [0, 5], "If": [0, 2, 3, 4, 5], "alreadi": [0, 5], "longer": 0, "actual": 0, "creat": [0, 1, 5, 6], "onc": [0, 1, 2, 7], "run": [0, 1, 4, 5, 6, 8], "core": [0, 5], "work": [0, 2, 3, 4, 5], "best": [0, 2, 5], "ndjson": [0, 3], "break": 0, "renam": [0, 2], "same": [0, 2, 3, 4, 5, 6], "wai": [0, 5, 7, 8], "old": [0, 5], "concaton": [0, 5], "datapackag": [0, 2, 5], "correct": [0, 5], "list": [0, 2, 4, 5], "escap": [0, 5], "quot": 0, "clearer": 0, "happen": 0, "rust": [0, 1, 2], "catch": 0, "certain": [0, 5], "mai": [0, 2, 5], "forign": 0, "decim": 0, "lower": [0, 5], "memori": [0, 2, 4], "index": 0, "foreign": [0, 2], "contraint": 0, "main_table_nam": [0, 5], "wa": [0, 2, 5], "number": [0, 2, 4, 5, 6], "except": [0, 2], "flatten": [0, 2, 4, 5, 7, 8], "correctli": 0, "function": [0, 7], "iter": [0, 4], "doc": [0, 7], "test": [0, 1], "iterator_flatten": 0, "deprec": 0, "just": [0, 5], "subset": 0, "lenient": 0, "tmp": 0, "delet": [0, 5], "preview": 0, "cli": 0, "librari": [0, 3], "save": [0, 5], "main": [0, 4], "yaglish": 0, "parser": [0, 2], "both": [0, 5], "schema_guess": 0, "tell": 0, "an": [0, 2, 4, 5], "empti": [0, 5], "do": [0, 2, 4, 5], "ctrlc": 0, "improv": [0, 5], "traceback": 0, "shown": 0, "occur": 0, "folder": [0, 5], "order": [0, 2, 5], "control": 0, "tab": [0, 5], "begin": 0, "handl": 0, "long": [0, 2, 3], "excel": [0, 5], "sheet": [0, 2, 5, 6], "see": [0, 2, 3, 4, 5], "github": [0, 1], "com": [0, 3, 5, 7], "kindli": 0, "issu": [0, 4, 5], "field_typ": [0, 2, 4, 5, 6], "redund": 0, "_link_": 0, "main_tabl": 0, "human": 0, "readabl": 0, "messag": [0, 5], "bad": 0, "charact": [0, 5], "strip": 0, "warn": [0, 4, 5], "check": [0, 5], "limit": [0, 6], "found": [0, 6], "lot": [0, 2, 5], "edg": 0, "instead": [0, 5], "leav": 0, "unus": 0, "partial": 0, "unwrap": 0, "channel": 0, "send": 0, "possibl": [0, 4], "reflect": 0, "lib": 0, "flatfil": 0, "new_with_dafualt": 0, "verbos": 0, "insta": 0, "mean": [0, 2, 5, 7], "onli": [0, 2, 4, 7], "show": [0, 4, 5], "specifi": [0, 2, 5, 7], "start": [0, 5], "root": [0, 5], "clippi": 0, "lint": 0, "metadata": [0, 2, 4, 6], "minor": 0, "speedup": 0, "format": [0, 2, 3, 7], "much": [0, 2], "abil": 0, "field_titl": [0, 2, 4, 5], "column": [0, 2, 5], "head": [0, 2, 5, 6], "titl": [0, 2, 4, 6], "out": [0, 2, 5], "jsonschema": [0, 5], "speed": [0, 5], "smartstr": 0, "smallvec": 0, "pypi": 0, "action": 0, "changelog": 0, "regress": 0, "anyhow": 0, "give": [0, 5], "context": 0, "prefix": 0, "namespec": 0, "separ": [0, 2], "_": [0, 2, 5], "them": 0, "load": [0, 2, 5, 6], "wheel": [0, 1, 2], "window": [0, 2, 8], "maco": [0, 2], "automat": [0, 8], "publish": 0, "inlin": 0, "One": 0, "item": [0, 4, 5], "treat": [0, 3, 5], "sub": [0, 2, 5], "pleas": 1, "contribut": 1, "stabl": 1, "toolchain": [1, 2], "virtualenv": 1, "clone": 1, "repo": 1, "virtual": 1, "ve": 1, "bin": 1, "activ": 1, "maturin": 1, "tool": [1, 2, 3, 4, 6], "extens": [1, 2, 5, 6], "pip": [1, 2], "releas": 1, "anylinux": [1, 2], "3": [1, 2, 4, 5, 6], "6": [1, 2, 4, 5], "distribut": 1, "sudo": 1, "docker": 1, "t": [1, 5], "To": [1, 8], "v": 1, "pwd": 1, "io": 1, "set": [1, 4, 5, 7, 8], "enviro": [1, 5], "flatterer_log": 1, "trace": 1, "full": [1, 3, 5], "debug": 1, "opinion": 2, "which": [2, 4, 5, 6, 7], "tri": 2, "analysi": [2, 7], "receiv": 2, "structur": [2, 5, 6], "deepli": 2, "nest": 2, "well": 2, "hard": 2, "determin": [2, 5], "what": [2, 5], "contain": [2, 4, 5, 6], "even": [2, 5], "after": [2, 5], "know": 2, "how": [2, 3], "part": 2, "pipelin": 2, "aim": 2, "first": [2, 4, 5], "go": [2, 4, 5, 6], "face": 2, "abov": [2, 3, 4, 5], "problem": 2, "It": [2, 4, 5, 6], "up": [2, 5], "your": [2, 4, 5], "hand": 2, "written": [2, 5], "benefit": 2, "over": [2, 5, 6], "most": [2, 5, 7], "approach": 2, "fast": 2, "bind": 2, "eas": 2, "10x": 2, "faster": 2, "effici": 2, "custom": 2, "practic": 2, "been": 2, "learnt": 2, "countless": 2, "gener": [2, 4, 5], "parent": [2, 5], "greater": 2, "binari": 2, "linux": 2, "x64": 2, "univers": 2, "x86": 2, "On": 2, "other": [2, 4, 5], "platform": [2, 4, 5, 6], "sai": [2, 5], "like": [2, 5, 6, 7], "game": [2, 3, 4, 5, 6], "1": [2, 4, 5, 6, 8], "A": [2, 4, 5, 6, 8], "released": [2, 4, 5, 6], "2015": [2, 4, 5], "01": [2, 4, 5], "xbox": 2, "playstat": 2, "rate": 2, "code": 2, "e": [2, 4, 5, 7], "everyon": [2, 4], "2": [2, 4, 5, 6, 7], "b": [2, 4, 5], "2016": [2, 4, 5], "pc": [2, 5], "games_dir": [2, 3, 4, 6], "refer": [2, 4], "detail": 2, "addit": [2, 5], "inform": [2, 4, 7], "configur": [2, 3, 8], "import": [2, 4, 5, 6], "By": [2, 5], "follow": [2, 5, 6, 7, 8], "tree": 2, "rating_cod": [2, 4, 6], "rating_nam": [2, 4, 6], "0": [2, 4, 5, 8], "special": 2, "primari": 2, "uniqu": [2, 5], "per": [2, 6], "promot": 2, "relationship": [2, 5, 6], "properti": [2, 5], "therefor": [2, 5], "_link_gam": [2, 5, 6], "everi": [2, 5], "easili": [2, 6], "would": [2, 5], "_link_platform": [2, 5], "_link__": 2, "table_nam": [2, 4, 5, 6], "referenc": 2, "about": [2, 4, 5, 6], "field_nam": [2, 4, 5, 6], "count": [2, 4, 5, 6], "text": [2, 4, 5, 6], "The": [2, 3, 4, 5, 6, 7], "initi": 2, "edit": [2, 5], "rerun": 2, "transform": 2, "new_games_dir": 2, "f": [2, 4, 5, 6], "myfield": [2, 5], "rearrang": [2, 5], "remov": [2, 5, 6], "flag": 2, "includ": [2, 5], "tabular": [2, 6], "datapackg": [2, 6], "spec": [2, 6], "retriev": 3, "varieti": 3, "below": 3, "exampl": [3, 4, 5], "thei": [3, 5], "rel": 3, "cat": [3, 6], "my": [3, 5], "bucketnam": 3, "bucket": [3, 7], "assum": [3, 5], "compress": 3, "select": [3, 5], "last": 3, "argument": [3, 4], "rest": 3, "games2": [3, 4], "games3": 3, "As": [3, 4, 5], "each": [3, 5, 6], "variou": 3, "For": [3, 4, 5], "new": [3, 6], "delimit": [3, 5], "repres": [4, 5, 6], "howev": [4, 5, 7], "dict": 4, "list_of_object": 4, "complic": [4, 5], "feed": 4, "ijson": 4, "def": 4, "array_item_gener": 4, "fixtur": 4, "basic": [4, 6], "rb": 4, "manipul": 4, "modifi": [4, 5], "yield": 4, "befor": [4, 5], "process": [4, 5], "byte": 4, "interpret": 4, "true": [4, 5, 6, 7], "describ": 4, "print": 4, "fiel": 4, "5": 4, "table_titl": [4, 5, 6], "develop": 4, "db": [4, 5, 6], "4": [4, 5], "7": 4, "_link_main": 4, "8": 4, "9": 4, "10": [4, 5], "11": 4, "similar": 4, "locat": 4, "fals": [4, 5], "locatin": 4, "dataset": [4, 5], "put": [4, 5, 6], "rating_co": 4, "system": [4, 5], "temporari": [4, 7], "g": [4, 5, 7], "look": [5, 6], "output_directori": 5, "base": 5, "nocsv": 5, "default": [5, 8], "noxlsx": 5, "nosqlit": 5, "noparquet": 5, "d": 5, "n": 5, "m": 5, "without": [5, 6, 7, 8], "p": 5, "j": 5, "o": 5, "l": 5, "singl": 5, "y": 5, "alwai": [5, 7], "": 5, "url": 5, "seper": 5, "denot": 5, "within": [5, 6], "h": 5, "given": 5, "slug": 5, "underscore_slug": 5, "w": 5, "integ": 5, "amount": [5, 6], "final": 5, "result": 5, "cpu": 5, "fit": 5, "exit": 5, "input_fil": 5, "user": [5, 6], "pass": 5, "host": [5, 6, 8], "dbname": 5, "should": 5, "In": 5, "env": 5, "database_url": 5, "my_env_var": 5, "envir": 5, "inputfil": 5, "ouput_dir": 5, "defin": [5, 7], "extra": 5, "appear": [5, 6], "ram": 5, "4gb": 5, "game_id": 5, "platforms_requir": 5, "platforms_nam": 5, "platforms_id": 5, "push": 5, "trust": 5, "suffici": 5, "back": 5, "togeth": 5, "analys": 5, "delimet": 5, "fastest": 5, "pars": 5, "jl": 5, "sometim": 5, "call": 5, "doe": [5, 6], "json_stream": 5, "goessner": 5, "net": 5, "articl": 5, "jsonpath": 5, "express": 5, "particular": 5, "evalu": 5, "against": 5, "null": 5, "film": 5, "second": 5, "care": 5, "json_path": 5, "logic": 5, "condit": 5, "either": 5, "OR": 5, "myschema": 5, "postgres_schema": 5, "differ": 5, "come": 5, "rule": 5, "conflict": 5, "coerc": 5, "origin": 5, "kept": 5, "still": 5, "id_prefix": 5, "random": 5, "ad": 5, "across": 5, "recommend": 5, "conatain": 5, "Not": 5, "complet": 5, "parallel": 5, "safe": 5, "two": 5, "try": 5, "currupt": 5, "retri": 5, "loos": 5, "databs": 5, "pre": 5, "chang": 5, "unwant": 5, "miss": 5, "match": 5, "eventu": 5, "easiest": 5, "sure": 5, "behaviour": 5, "exclud": 5, "only_field": 5, "tables_nam": 5, "basi": 5, "tabless": 5, "only_t": 5, "ever": 5, "associ": 5, "With": 5, "under": 5, "inline_one_to_on": 5, "encod": 5, "remot": 5, "server": [5, 6], "otherwis": 5, "mytabl": 5, "myprefix_": 5, "myprefix_myt": 5, "namespac": 5, "table_prefix": 5, "myobject_myfield": 5, "myobject": 5, "theres": 5, "my_object": 5, "my_field": 5, "my_object_my_field": 5, "confus": 5, "might": 5, "expect": 5, "clash": 5, "fix": 5, "whatev": 5, "___": 5, "my_object__my_field": 5, "__": 5, "path_separo": 5, "take": 5, "modif": 5, "slugifi": 5, "charactor": 5, "alphanumer": 5, "replac": 5, "strang": 5, "turn": 5, "previou": 5, "my_strange_titl": 5, "left": 5, "schema_titl": 5, "underscore_slug_": 5, "done": 5, "x3": 5, "x1": 5, "24": 5, "worth": 5, "lead": [5, 7], "perform": 5, "veri": 5, "100": 5, "slower": 5, "ensur": 5, "circumst": 5, "wrong": 5, "chosen": 5, "reduc": 5, "sacrif": 5, "int": 6, "unless": [6, 7], "help": 6, "definit": 6, "numer": 6, "timestamp": 6, "psql": 6, "someth": 6, "password": 6, "adapt": 6, "plain": 6, "advantag": 6, "games_dir_": 6, "sqlite3": 6, "my_databas": 6, "content": 6, "skip": 6, "profil": 6, "packag": 6, "resourc": 6, "primarykei": 6, "fetch": 7, "provid": 7, "aws_default_region": 7, "u": 7, "east": 7, "aws_access_key_id": 7, "access_key_id": 7, "aws_secret_access_kei": 7, "secret_access_kei": 7, "aws_endpoint": 7, "endpoint": 7, "aws_session_token": 7, "token": 7, "aws_container_credentials_relative_uri": 7, "aw": 7, "amazon": 7, "amazonec": 7, "developerguid": 7, "task": 7, "iam": 7, "role": 7, "html": 7, "aws_allow_http": 7, "permit": 7, "tl": 7, "commonli": 7, "alon": 7, "upload": 7, "someaccesskeyid": 7, "somesecretaccesskeyid": 7, "mybucket": 7, "mydata": 7, "flattenedoutput": 7, "store": 7, "constrain": 7, "serverless": 7, "read": 7, "twice": 7, "known": 7, "upfront": 7, "learn": 8, "visual": 8, "feedback": 8, "webserv": 8, "port": 8, "8080": 8, "localhost": 8, "127": 8, "max_siz": 8, "size": 8, "mb": 8, "maximum": 8, "500": 8, "no_brows": 8, "browser": 8}, "objects": {}, "objtypes": {}, "objnames": {}, "titleterms": {"chang": 0, "log": [0, 1], "0": 0, "19": 0, "17": 0, "2024": 0, "06": 0, "18": 0, "new": [0, 5], "fix": 0, "15": 0, "05": 0, "09": 0, "14": 0, "02": 0, "10": 0, "13": 0, "01": 0, "26": 0, "12": 0, "2023": 0, "03": 0, "8": 0, "21": 0, "6": 0, "5": 0, "07": 0, "4": 0, "3": 0, "11": 0, "1": 0, "2022": 0, "16": 0, "2": 0, "25": 0, "27": 0, "23": 0, "22": 0, "remov": 0, "24": 0, "04": 0, "28": 0, "9": 0, "7": 0, "2021": 0, "ad": 0, "20": 0, "develop": 1, "local": [1, 2, 3], "instal": [1, 2], "build": 1, "flatter": 2, "make": 2, "json": [2, 5, 6], "introduct": 2, "rational": 2, "exampl": [2, 7], "run": 2, "cli": [2, 5], "web": [2, 8], "interfac": [2, 8], "As": 2, "python": [2, 4, 5], "librari": [2, 4], "output": [2, 4, 5, 6], "file": [2, 3, 5, 6], "main": [2, 5], "tabl": [2, 5, 6], "One": [2, 5], "To": [2, 5], "mani": 2, "link": [2, 5], "field": [2, 5, 6], "csv": [2, 6], "input": 3, "sourc": 3, "stdin": 3, "http": 3, "": 3, "s3": [3, 7], "gzip": 3, "multipl": 3, "usag": [4, 5, 7], "creat": 4, "panda": 4, "datafram": 4, "option": 5, "refer": 5, "help": 5, "small": 5, "onli": 5, "format": 5, "name": 5, "pushdown": 5, "No": 5, "path": 5, "arrai": 5, "line": 5, "delemit": 5, "ndjson": 5, "stream": 5, "filter": 5, "forc": 5, "postgr": [5, 6], "schema": 5, "evolv": 5, "drop": 5, "truncat": 5, "inlin": 5, "prefix": 5, "separ": 5, "titl": 5, "stat": 5, "preview": 5, "thread": 5, "sql": [5, 6], "script": 5, "low": 5, "memori": 5, "api": 5, "xlsx": 6, "sqlite": 6, "parquet": 6, "postgresql": 6, "postgresql_schema": 6, "postgresql_load": 6, "sqlite_schema": 6, "sqlite_load": 6, "datapackag": 6, "url": 7, "region": 7, "authent": 7, "note": 7}, "envversion": {"sphinx.domains.c": 2, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 8, "sphinx.domains.index": 1, "sphinx.domains.javascript": 2, "sphinx.domains.math": 2, "sphinx.domains.python": 3, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx": 57}, "alltitles": {"Change Log": [[0, "change-log"]], "[0.19.17] - 2024-06-18": [[0, "id1"]], "New": [[0, "new"], [0, "id9"], [0, "id11"], [0, "id13"], [0, "id23"], [0, "id25"], [0, "id29"], [0, "id32"], [0, "id38"], [0, "id40"], [0, "id42"], [0, "id46"], [0, "id48"], [0, "id51"], [0, "id74"], [0, "id76"], [0, "id79"], [0, "id83"], [0, "id89"]], "Fixed": [[0, "fixed"], [0, "id3"], [0, "id5"], [0, "id7"], [0, "id15"], [0, "id17"], [0, "id19"], [0, "id21"], [0, "id27"], [0, "id30"], [0, "id34"], [0, "id36"], [0, "id44"], [0, "id58"], [0, "id60"], [0, "id64"], [0, "id66"], [0, "id70"], [0, "id72"], [0, "id77"], [0, "id81"], [0, "id85"], [0, "id87"], [0, "id96"], [0, "id98"]], "[0.19.15] - 2024-05-09": [[0, "id2"]], "[0.19.14] - 2024-02-10": [[0, "id4"]], "[0.19.13] - 2024-01-26": [[0, "id6"]], "[0.19.12] - 2023-12-03": [[0, "id8"]], "[0.19.10] - 2023-10-02": [[0, "id10"]], "[0.19.8] - 2023-06-21": [[0, "id12"]], "[0.19.6] - 2023-06-21": [[0, "id14"]], "[0.19.5] - 2023-06-07": [[0, "id16"]], "[0.19.4] - 2023-05-26": [[0, "id18"]], "[0.19.3] - 2023-05-11": [[0, "id20"]], "[0.19.1] - 2023-05-10": [[0, "id22"]], "[0.18.0] - 2023-03-17": [[0, "id24"]], "[0.17.1] - 2023-01-07": [[0, "id26"]], "[0.17.0] - 2022-12-03": [[0, "id28"]], "[0.16.2] - 2022-10-25": [[0, "id31"]], "[0.16.2] - 2022-10-16": [[0, "id33"]], "[0.16.1] - 2022-10-16": [[0, "id35"]], "[0.16.0] - 2022-07-27": [[0, "id37"]], "[0.15.0] - 2022-07-27": [[0, "id39"]], "Changed": [[0, "changed"], [0, "id52"], [0, "id54"], [0, "id56"], [0, "id62"], [0, "id68"], [0, "id80"], [0, "id84"], [0, "id88"], [0, "id91"], [0, "id93"], [0, "id100"]], "[0.14.2] - 2022-07-27": [[0, "id41"]], "[0.14.1] - 2022-07-23": [[0, "id43"]], "[0.14.0] - 2022-07-22": [[0, "id45"]], "Removed": [[0, "removed"]], "[0.13.2] - 2022-06-24": [[0, "id47"]], "[0.13.1] - 2022-05-27": [[0, "id49"]], "[0.13.0] - 2022-04-28": [[0, "id50"]], "[0.12.12] - 2022-04-15": [[0, "id53"]], "[0.12.11] - 2022-04-09": [[0, "id55"]], "[0.12.10] - 2022-03-21": [[0, "id57"]], "[0.12.9] - 2022-03-07": [[0, "id59"]], "[0.12.8] - 2022-03-01": [[0, "id61"]], "[0.12.7] - 2022-02-27": [[0, "id63"]], "[0.12.6] - 2022-02-26": [[0, "id65"]], "[0.12.4] - 2022-02-18": [[0, "id67"]], "[0.12.3] - 2022-02-15": [[0, "id69"]], "[0.12.1] - 2022-02-15": [[0, "id71"]], "[0.12] - 2022-02-02": [[0, "id73"]], "[0.11] - 2022-01-25": [[0, "id75"]], "[0.10.1] - 2021-12-22": [[0, "id78"]], "[0.10] - 2021-12-22": [[0, "id82"]], "[0.9] - 2021-12-12": [[0, "id86"]], "[0.8] - 2021-12-01": [[0, "id90"]], "[0.7.1] - 2021-12-01": [[0, "id92"]], "[0.7] - 2021-11-24": [[0, "id94"]], "Added": [[0, "added"], [0, "id102"], [0, "id104"], [0, "id106"]], "[0.6.2] - 2021-11-24": [[0, "id95"]], "[0.6.1] - 2021-11-24": [[0, "id97"]], "[0.6] - 2021-11-23": [[0, "id99"]], "[0.5] - 2021-11-22": [[0, "id101"]], "[0.4] - 2021-11-20": [[0, "id103"]], "[0.3] - 2021-10-25": [[0, "id105"]], "Development": [[1, "development"]], "Local install": [[1, "local-install"]], "Builds": [[1, "builds"]], "Logging": [[1, "logging"]], "Flatterer. Making JSON flatterer": [[2, "flatterer-making-json-flatterer"]], "Introduction": [[2, "introduction"]], "Rationale": [[2, "rationale"]], "Install": [[2, "install"]], "Example JSON": [[2, "example-json"]], "Running Flatterer": [[2, "running-flatterer"]], "CLI": [[2, "cli"]], "Local web interface": [[2, "local-web-interface"]], "As python library": [[2, "as-python-library"]], "Output Files": [[2, "output-files"]], "Main Table": [[2, "main-table"]], "One To Many Table": [[2, "one-to-many-table"]], "Link Fields": [[2, "link-fields"]], "Fields CSV": [[2, "fields-csv"]], "Input Sources": [[3, "input-sources"]], "Local File": [[3, "local-file"]], "Stdin": [[3, "stdin"]], "HTTP(s)": [[3, "http-s"]], "S3": [[3, "s3"], [7, "s3"]], "GZIP": [[3, "gzip"]], "Multiple inputs": [[3, "multiple-inputs"]], "Python Library": [[4, "python-library"]], "Usage": [[4, "usage"]], "Output": [[4, "output"]], "Creating pandas DataFrames": [[4, "creating-pandas-dataframes"]], "Option Reference": [[5, "option-reference"]], "Help (CLI Only) ": [[5, "help-small-cli-only-small"]], "Output Formats": [[5, "output-formats"]], "CLI Usage": [[5, "cli-usage"], [5, "id1"], [5, "id3"], [5, "id5"], [5, "id7"], [5, "id9"], [5, "id11"], [5, "id13"], [5, "id15"], [5, "id17"], [5, "id19"], [5, "id21"], [5, "id23"], [5, "id25"], [5, "id27"], [5, "id29"], [5, "id31"], [5, "id33"], [5, "id35"], [5, "id37"], [5, "id39"], [5, "id41"], [5, "id43"], [5, "id45"], [5, "id47"], [5, "id49"]], "Python Usage": [[5, "python-usage"], [5, "id2"], [5, "id4"], [5, "id6"], [5, "id8"], [5, "id10"], [5, "id12"], [5, "id14"], [5, "id16"], [5, "id18"], [5, "id20"], [5, "id22"], [5, "id24"], [5, "id26"], [5, "id28"], [5, "id30"], [5, "id32"], [5, "id34"], [5, "id36"], [5, "id38"], [5, "id40"], [5, "id42"], [5, "id44"], [5, "id46"], [5, "id48"], [5, "id50"], [5, "id51"], [5, "id52"]], "Main Table Name": [[5, "main-table-name"]], "Pushdown Fields": [[5, "pushdown-fields"]], "No Link Fields": [[5, "no-link-fields"]], "Path to JSON Array": [[5, "path-to-json-array"]], "New Line Delemited JSON (NDJSON)": [[5, "new-line-delemited-json-ndjson"]], "JSON Stream": [[5, "json-stream"]], "JSON Path Filter": [[5, "json-path-filter"]], "Force": [[5, "force"]], "Postgres Schema": [[5, "postgres-schema"]], "Evolve Tables": [[5, "evolve-tables"]], "Drop Tables": [[5, "drop-tables"]], "Truncate Tables": [[5, "truncate-tables"]], "Fields File": [[5, "fields-file"]], "Only Fields": [[5, "only-fields"]], "Tables File": [[5, "tables-file"]], "Only Tables": [[5, "only-tables"]], "Inline One To One": [[5, "inline-one-to-one"]], "Arrays as Table": [[5, "arrays-as-table"]], "Schema": [[5, "schema"]], "Table Prefix": [[5, "table-prefix"]], "Path Separator": [[5, "path-separator"]], "Schema Titles": [[5, "schema-titles"]], "Stats": [[5, "stats"]], "Preview": [[5, "preview"]], "Threads": [[5, "threads"]], "Sql Scripts": [[5, "sql-scripts"]], "Low Memory (api only)": [[5, "low-memory-api-only"]], "Outputs": [[6, "outputs"]], "CSV": [[6, "csv"]], "XLSX": [[6, "xlsx"]], "SQLITE": [[6, "sqlite"]], "POSTGRES": [[6, "postgres"]], "PARQUET": [[6, "parquet"]], "fields.csv": [[6, "fields-csv"]], "tables.csv": [[6, "tables-csv"]], "Postgresql Files": [[6, "postgresql-files"]], "postgresql_schema.sql": [[6, "postgresql-schema-sql"]], "postgresql_load.sql": [[6, "postgresql-load-sql"]], "Sqlite Files.": [[6, "sqlite-files"]], "sqlite_schema.sql": [[6, "sqlite-schema-sql"]], "sqlite_load.sql": [[6, "sqlite-load-sql"]], "datapackage.json": [[6, "datapackage-json"]], "URL": [[7, "url"]], "Region and authentication.": [[7, "region-and-authentication"]], "Example": [[7, "example"]], "Usage Note": [[7, "usage-note"]], "Web Interface": [[8, "web-interface"]]}, "indexentries": {}}) \ No newline at end of file