I'm using postgres 10.5

Fixed it :)

no length arg

yes )

Does Pony have any length limits, or is it whatever the database supports?

For CLOBs? No, only database provider restrictions

great :)

Hi Alexander, is there any way to perform batch insert or update? And especially in raw SQL? thanks.

Hi, I’m playing around with pony and tornado and it’s gone really nicely so far, pony is really quite awesome. I’ve hit a snafoo though, I can’t seem to perform “Entity.to_json()” conversions as I haven’t setup user access control the pony docs don’t mention this scenario. I was wondering if there was an example or doc that I could read up on. Most of my searches end up with info about actual tornadoes.

use to_dict instead

Yep, then you have a normal dict which tornado can handle how it wishes

Is there a way I can make pony show the connection string it is using to connect to postgres?

some internal private value it is fine, it is only for debugging something else

Right, thanks, I'll switch to dict then, for reference I did manage to find these pages https://docs.ponyorm.com/ponyjs_security.html and https://stackoverflow.com/q/41030262. I also tried importing the Pony.flask module in the hopes that it might discuss how to get this working but I get "Module not found" instead, presumably this is deprecated ?

ponyjs is javascript frontend stuff

sorry I think I'm wrong

Ok, do you know if there is more documentation for the PonyJS stuff ?

I don't know I'm afraid. I'd suggest controlling access in your web framework

For the connection infor have you tried enabling the database debug switch, it shows all the connections/disconnections to SQLite, perhaps it does the same for Postgres (I haven't tried with a real db yet) "set_sql_debug(True)"

Thanks Mr. Bell, I'll code dig this afternoon then :D and use dicts for the interim.

cool. I tried sql_debug but unfortunately it didn't work

Can you wireshark it ?

Maybe

Does pony connect to postgres via the unix socket by default?

Have you installed psycopg2 ?

I solved the problem

Python was working fine, because it was using unix socket

other program was connecting over TCP so needed a password

Hi Alexander, is there any way to perform batch insert or update? And especially in raw SQL? thanks.

Hi Windfarer, at this moment Pony does not provide batch inserts or updates, you can get native DBAPI connection using db.get_connection() and manually perform cursor = connection.cursor() cursor.executemany(custom_sql, list_of_args) In PostgreSQL you need to perform psycopg2.extras.execute_batch / execute_values instead of executemany in order to get good performance

Hi Carel, PonyJS remains unreleased. We plan to rewrite PonyJS but at this moment have no resources for that

At this moment you need to use to_dict, I hope later we can provide something better

Hi Windfarer, at this moment Pony does not provide batch inserts or updates, you can get native DBAPI connection using db.get_connection() and manually perform cursor = connection.cursor() cursor.executemany(custom_sql, list_of_args) In PostgreSQL you need to perform psycopg2.extras.execute_batch / execute_values instead of executemany in order to get good performance

thank you?

At this moment you need to use to_dict, I hope later we can provide something better

Ok, so I’ll skip to_json for now. Thank you for your response. It’s a really nice api by the way. Also is the PonyJS mixed in with the Python code at the moment ? Or is it hosted else where ? I’ll pull the repo in a moment and have a poke around.

For getting json representation from my models I have used also the Marshmallow library. Maybe worth looking into

Is there a way I can make pony show the connection string it is using to connect to postgres?

Pony does not uses connection string internally, it passes to psycopg2 argsuments that we provided in Database constructor or db.bind. You can access them in the following undocumented way: print(db.provider.pool.args, db.provider.pool.kwargs)

For getting json representation from my models I have used also the Marshmallow library. Maybe worth looking into

Thanks I’ll check it out :)

PonyJS JavaScript file is in the repository, but not included in PyPI distributive, becuase of it unreleased state. Python PonyORM modules has to_json, but it is undocumented

Some time later we will remove to_json and replace it with some new API

One last question, does the permissions_for function fall under the PonyJS code ?

yes

Ok, thanks :)

Hi, there is any way to have the database models on a separated file? There is an example?

https://stackoverflow.com/questions/51601838/ponyorm-multiple-model-files/51671065#51671065

Oh, thank you

Sure

Hello, sorry to bother you! I’m interested in helping out with pony, but it seems like there isn’t a CONTRIBUTING.md or many merge requests by other people. I’m guessing you’re limiting contributions to employees or something? I just want to make sure before I accidentally do something stupid.

Hi Shane! Glad you want to help! We are not limiting contributions to employees, but if a contribution is non-trivial we ask a user to sign a contributor agreement, just to be sure that the contributed code can be truly licensed under MIT/Apache 2, and doesn't contain some legal/patent-related surprises. It is a common practice, Django does the same. Saying that, it may be not an easy task to have Pony pool request accepted. Pony internals are pretty complex, and many pool requests does not implement all necessary code in a correct way. For example, adding a new data type requires not only writing appropriate converter to/from database values, but also adding corresponding monads to translate new types of expressions inside generators. Because of that, we can accept a pull request only after it was reviewed carefully by people who understands all Pony internal details. I very often was in situation when some feature looks simple, but during implementing it it gradually become clear that big additional work is necessary in caching mechanism, nested subqueries or somewhere else. So, the first impression can be deceptive. If you want to write some contribution I recommend you to initially discuss your ideas here so we can roughly estimate complexity of the task and give some advices. After the pull request is done, sometimes it worth to remind us about it, because we can be distracted with other tasks

I could help out on some of the tasks that are still outstanding from https://github.com/ponyorm/pony/issues/184#issuecomment-257295870, although that post is around 2 years old so maybe you've gotten those taken care of. I use this library a lot at work and understanding how the source code works is important. My main concern is that the source is difficult to understand and debug at certain points (eg core.py) so I would like to help with fixing that or documenting the innards more thoroughly. I hope that you don't take this the wrong way! I understand that this isn't a trivial task so I'll probably just be helping out with the backlog if I do end up helping out.

This would be great, it is always worrying to dive into the inner parts of ponyOrm

Sorry, my phone is at 1%

Splitting core.py and maybe autoformat (black?) seems a good start to me. Commenting is not an easy thing but would be very helpfull

Maybe I could start a small proof of concept branch for just core.py and start from there? I don’t think much of the stated blocking features would touch it so there won’t be too bad of a merge conflict

And if it pans out I keep it as a parallel branch until it’s fully baked, if it isn’t then I just delete the branch

In principle I'm for refactoring to PEP8. The only deviation I want to keep is to always name a variable according to its type, that is, use attr instead of self, etc. That make full-project searching of attr.something much easier, especially if there is entity.something, table.something, etc. But all other things may be changed according to PEP8. The reason why we didn't fix it yet is we have some long-standing private branches which were not merged into master yet. And if we reformat code in master, then we would not be able to do merge anymore, because of huuuge and totally unreadable merge conflicts after reformatting. I hope we can merge these branches this year, and then will be able to reformat the code

Is it possible to contribute to those branches or is it just not something you want to share before it’s ready? If it’s the latter then I understand and I’ll wait until they’re ready to be released.

I don't fully reviewed the code in these branches yet. Some things requires changes, but at this moment it is hard to say what exactly. Some time ago I came to the conclusion that it is very dangerous to accept code changes in Pony that I do not fully understand, because it can slow down further refactoring tremendously. I suggest that we return to this topic later. I hope I will have time to review these branches during this year

That makes sense, thanks. I guess I’ll check back in later.

I believe we have found a minor race condition source in Pony. Our codebase (Tribler-IPv8-Trustchain project) uses nosetests for testing, twisted for asynchronous programming and libtorrent for torrent downloads. When we run a nosetests-managed test that downloads something via Libtorrent, we get an exception in thread Twisted on a Python interpreter exit. I believe this happens because Pony's db.disconnect() does not have enough time to stop the connection before the interpreter is torn down . If sleep(1) is added after db.disconnect() the exception is much less likely to happen. It is interesting to note that running the same test through trial never crashes the interpreter. Before moving to Pony we were using our own SQLite manager based on APSW. It never showed any signs of race conditions, probably because APSW was designed to be thread-safe: https://www.reddit.com/r/Python/comments/yx20w/the_sqlite_lock_timeout_nightmare/ We would like to move our whole codebase to Pony. And that means hundreds of connections per second on a very large and diverse user base (Trustchain was chosen by the Government of Netherlands to become the base platform for the new Dutch passport system). Rock-stable performance is crucial for us. @metaprogrammer , could you please comment on Pony stability under heavy concurrent load, with asynchronous calls etc? Is it possible to move to APSW as a SQLite provider of choice for thread safety? Thanks.

Hi Grigory! So, the project in which an exception happens after db.disconnect() uses Pony with SQLite?

I believe this exception is probably not related to Pony or SQLite. Can you provide the exception text?

Pony designed to be thread-safe when each db_session is used from a single thread. sqlite3 module used for connection to SQLite database should be thread-safe too. The famous sqlite3 lock timeout problem is caused by two things, first is that SQLite uses a database-level write lock instead of more granular row-level locks, and the second is that by default standard sqlite3 connection set this lock too early. Pony mitigates the second problem by controlling the lock and setting it as later as possible. It is thread-safe, but lower transaction isolation level from "serializable" to "read committed" (just as in PostgreSQL)

I think moving to APSW will not improve concurrency, because the first problem (database-level lock) remains in APSW, and the second problem (too aggressive locking n sqlite3) was already solved by Pony.

Anyway, it is possible to add APSW support in Pony. But it is a medium-sized task which may require, say, two weeks, and at this moment we don't have time and budget for it. Another tasks looks more important. If someone are ready to sponsor adding APSW support, we can change priorities.

I have started my vacation today, and I'm going to Greece, where I will be on a sailing yacht with a minimal Internet. I will be able to return to active development starting from the September 3

Returning to you question, db.disconnect() with sqlite3 should basically clear internal SQLite buffers and close file descriptor. The should be no anything asynchronous in it.

@metaprogrammer , thank you for an extensive answer! We'll look into the problem further on our own. Goodbye and have a wonderful vacation!

Nice holidays !!

Thanks!

Hi!

First all sorry for my english! I have a question: supposed I have a table called ‘person’ with a INT field called ’age’

So in my python script i create the class:

class Person(db.Entity): age = Required(str)

(so I declare voluntarily age with string type)

So i execute the command “db.generate_mapping(check_tables=True)”

Why I have not errors with this code?

I have errors only if I make a commit

(supposed i create an istance of Person object: p = Person(age=“t24”)…the object is created but commit fails)

I would like receive an error when I declare “age = Required(str)” because the real type of ‘age’ field is INT. According to check_tables role: “check_tables (bool) – when True, Pony makes a simple check that the table names and attribute names in the database correspond to entities declaration. It doesn’t catch situations when the table has extra columns or when the type of particular column doesn’t match. ”

Thanks!

Hmm.. so I’m not entirely certain but I think that check would only happen if you have “field = Required(int)”. That is pony would not check the type in the underlying database if the table already exists. There is a flag to indicate if a table already exist, perhaps setting that might change things for you. Also if you’re using SQLite it stores everything as whatever data type you give it so the check wouldn’t matter. On other stricter databases this would matter and might throw an error.