Ah, seems like I cannot select like this because its type is (<class 'str'>, <class 'int'>)

what do i need to do in order to make objects with a UUID attribute json serializable? currently it fails with: TypeError: keys must be str, int, float, bool or None, not UUID

if i do json.dumps myself it will be double-encoded

Yes, if it is the most outer db_session

I suspect there is another db_session which is too wide in scope, like, wraps all your application

(update interval of the chart: 1-2 seconds) I have a problem where I’m going though my database to find stuff which needs updating. Apparently pony isn’t good for that as it quickly grows to ~15GB of memory, bringing down my server. How can I improve memory footprint?

Hi! I'm writing a Python library to work with ConceptNet (http://conceptnet.io/) offline. There is an existing library from ConceptNet creators, but it requires PostgreSQL to be installed, which is not convenient for linguists. So, I rely on SQLite+Pony. I'm loading the dump by creating ORM objects. The problem is that there are 34074917 edges in the dump. Loading 100000 edges took 7 minutes. Here is my PC info: Operating System: openSUSE Tumbleweed 20190814 Kernel Version: 5.2.8-1-default Processors: 4 × Intel® Core™ i5-3320M CPU @ 2.60GHz Memory: 15,3 GiB of RAM 300 GiB SSD drive By extrapolating I got 39 hours to load the dump. That's too much. Is there a faster way? For example, store everything in memory, and save DB after the filling?

You should not load all objects to test them in memory. Try to formulate your query to retrieve only objects which need updating

Hi Roman! I'm not exactly understand what requires so much time. What do you mean by "load the dump", is it inserting new rows to the database, or reading data from database to memory?

Objects are unloaded after db_session exit, but if you still hold reference to one of them in some global variable, it will hold other objects as well, because objects are interrelated via relationships. So, if you work with different db_sessions and does not hold references to objects in global variables, they should be unloaded automatically. You can also use option db_session(strict=True) to force object unloading even if you hold some reference in a global variable

Objects are unloaded after db_session exit, but if you still hold reference to one of them in some global variable, it will hold other objects as well, because objects are interrelated via relationships. So, if you work with different db_sessions and does not hold references to objects in global variables, they should be unloaded automatically. You can also use option db_session(strict=True) to force object unloading even if you hold some reference in a global variable

How can I make sure that there are no db_session already active?

Hi @metaprogrammer. Here is the example of a row from CSV file: /a/[/r/Antonym/,/c/ab/агыруа/n/,/c/ab/аҧсуа/] /r/Antonym /c/ab/агыруа/n /c/ab/аҧсуа {"dataset": "/d/wiktionary/en", "license": "cc:by-sa/4.0", "sources": [{"contributor": "/s/resource/wiktionary/en", "process": "/s/process/wikiparsec/1"}], "weight": 1.0} Here is the ER diagram for my library: https://editor.ponyorm.com/user/iqt/conceptnet_lite/designer Here is the page, where you can download the gzipped CSV dump: https://github.com/commonsense/conceptnet5/wiki/Downloads I read file line by line using standart CSV module and create the objects in a for loop.

You can avoid creating object in memory while populating database from CSV file by using db.insert method: with db_session: ... for a, b, c in some_data: id = db.insert(table_name, column1=a, column2=b, column3=c) ... to be even more efficient, you can use low-level database connection: # for postgresql replace ? to %s sql = "INSERT INTO my_table (column1, column2, column3) values(?, ?, ?)" with db_session: connection = db.get_connection() cursor = connection.cursor() for a, b, c in some_data: cursor.execute(sql, (a, b, c)) or # for postgresql replace ? to %s sql = "INSERT INTO my_table (column1, column2, column3) values(?, ?, ?)" with db_session: connection = db.get_connection() cursor = connection.cursor() params = [] for a, b, c in some_data: params.append((a, b, c)) cursor.executemany(sql, params)

You can avoid creating object in memory while populating database from CSV file by using db.insert method: with db_session: ... for a, b, c in some_data: id = db.insert(table_name, column1=a, column2=b, column3=c) ... to be even more efficient, you can use low-level database connection: # for postgresql replace ? to %s sql = "INSERT INTO my_table (column1, column2, column3) values(?, ?, ?)" with db_session: connection = db.get_connection() cursor = connection.cursor() for a, b, c in some_data: cursor.execute(sql, (a, b, c)) or # for postgresql replace ? to %s sql = "INSERT INTO my_table (column1, column2, column3) values(?, ?, ?)" with db_session: connection = db.get_connection() cursor = connection.cursor() params = [] for a, b, c in some_data: params.append((a, b, c)) cursor.executemany(sql, params)

Generate it once and let users download it rather than force them to generate it?