Headaches when migrating to Connexion v3 (flask): different threads are killing me

[FelixSchwarz]

Preface: This is just my personal experience and I don't want to be ungrateful at all/blame on Connexion developers. I'd love to be wrong, so if you have any suggestions, feel free to share.

Back in the Connexion v2 days, we implemented a small rest service with flask. This went pretty well, not much different from any other (synchronous) flask app we wrote. Then it seemed like Connexion would be another dead project with almost no activity so we postponed migration to Connexion 3.

Since the development activities in the last month increased, I decided to take a stab at migrating that simple rest service to Connexion 3 with the idea of removing technical debt (unsupported legacy versions) but I have to say that Connexion 3 caused a lot of headaches.

My idea is/was to continue using synchronous flask handlers, basically just upgrade Connexion and do the minimum amount of changes so I can use the latest version.

different threads/problems with in-memory sqlite

The main problem is that Connexion runs synchronous handlers in another thread. This is a major problem for my test-suite which uses in-memory sqlite. Now sqlalchemy+sqlite do not like passing these instances between threads which is a major problem.

I'd love to see a way to disable threads for tests.

security middleware has no flask context

We do check API keys against a database. Previously the DB session was set up in a flask handler similar to what other flask-based apps are doing. However now flask's handlers run only after the security middleware is executed. This means we need a completely different DB setup code which is annoying.

[chrisinmtown]

I just upgraded my backing store aka data service from connexion v2 to v3. I followed /exactly/ your strategy of

..continue using synchronous flask handlers, basically just upgrade Connexion and do the minimum amount of changes so I can use the latest version.

My project uses SQLAlchemy & Marshmallow; also the helpers Flask-SQLAlchemy and marshmallow-sqlalchemy. For tox tests, I configure a SQLAlchemy URL with "sqlite:///" and then all the REST endpoints do their work on an in-memory database, so I don't have to mock anything at all, that is nice. I had to create a test fixture that creates all the tables. To the best of my knowledge, I didn't enable (nor disable) /anything/ about Connexion threads.

I can share some details but I can't just post the project, my employer would be peeved.

[FelixSchwarz]

Ah, that sounds really helpful.

I think the root of the problem in our case is

rw_engine = create_engine('sqlite:///:memory:')
rw_factory = sessionmaker(bind=rw_engine, autoflush=False)
session scoped_session(rw_factory, scopefunc=threading.get_ident)

I am a bit curious how you can create the tables in the test code and then use the same DB connection in the tested server functions.

[chrisinmtown]

Here's how I create the SQLAlchemy object. Please note the Flask app config has key SQLALCHEMY_DATABASE_URI and for dev/test the value is sqlite:///:memory:

from flask_sqlalchemy import SQLAlchemy
from flask_marshmallow import Marshmallow

def create_db(app: FlaskApp) -> tuple:
    db = SQLAlchemy(app)
    orm_ma = Marshmallow(app)
    return db, orm_ma

My app has no code for creating an engine, session maker, scoped session, etc.

Here's the test fixture that creates a fresh database:

@pytest.fixture
def database() -> None:
    with conn_app.app.app_context():
        db.create_all()
        build_db.create_sim_db()
        db.session.commit()

That one extra function create_sim_db adds a bunch of fake-data rows to tables by using the model classes that inherit from db.Model, like this:

model = MyModel(attr1="one", attr2="two")
db.session.add(model)
db.session.commit()

[FelixSchwarz]

Thank you for pointing me to flask_sqlalchemy. We are not using that as we needed some extra features but a small dummy app with flask_sqlalchemy showed that their implementation works.

When I read their code I discovered that they use create_engine(..., connect_args={'check_same_thread': False}) for in-memory sqlite databases. Basically there is a way to just disable the check on the sqlite side which solves my issue. :-)