Maximum call stack size exceeded #10449
Comments
Object.values would return the values in the key value pair so instead of returning Please provide a script that demonstrates these issues you have |
IslandRhythms
added
the
needs repro script
|
I can comment on this. The cursor doesn't clean up after itself, so if you have millions of data points, it will eventually use up too much:
Here is a simplified version of my script: const fs = require('fs')
const mongoose = require('mongoose')
const log = require('single-line-log').stdout
const PlacePointModel = require('./schema/PlacePoint.js').PlacePointModel
let total = 0
async function BUILD () {
await mongoose.connect('mongodb://127.0.0.1:27017/data', {
useNewUrlParser: true,
useUnifiedTopology: true,
useFindAndModify: false,
useCreateIndex: true
})
const db = mongoose.connection
const pointCursor = PlacePointModel.find().populate('relations').cursor()
for (let point = await pointCursor.next(); point != null; point = await pointCursor.next()) {
log(++total)
// pull
const { ... } = point
// manipulate
...
// store
fs.appendFileSync('./data.json', JSON.stringify(feature) + '\n')
}
db.close()
}
BUILD()version of node: 12.22.3 |
Sorry, I meant Object.entries, but the main culprit here is that it reported values/keys/content which is not part of the original object. Simplified version of a scirpt that produces this errornous output: import 'reflect-metadata';
import { DBUserModel } from '@hokify/user-data';
import { DBMailBounceModel } from '@hokify/messaging-servicefeedback';
import { ObjectId } from 'mongodb';
import { MongooseConnection } from '../src';
(async () => {
await MongooseConnection.connect();
// fetch a user object
const selUser = await DBUserModel.findOne({
_id: new ObjectId('60e61c0dd8c15500190e031b')
}).exec();
// fetc mail bounces
await DBMailBounceModel.findOne({ email: selUser.general.email })
.exec()
.then(bounce => {
selUser.mailBounce = bounce;
});
console.log('selUser.mailBounce', selUser.mailBounce);
console.log('selUser.privacy', selUser.privacy);
console.log('JSON.stringify(selUser.privacy)', JSON.stringify(selUser.privacy));
console.log('Object.keys(selUser.privacy)', Object.keys(selUser.privacy));
console.log(
'Object.entries(selUser.privacy)',
Object.entries(selUser.privacy).map(
([key, _value]) => `privacy[${key}]` + `: ${selUser.privacy[key] ? 'Exists' : ' FAIL'}`
)
);
})();and the output of this scirpt: look at selUser.privacy and JSON.stringify => all good there is no such thing as mailBounce.. furthermore if I add .lean() to the user query, all works correctly. Version checks:
so it seems the bug got introduced in 5.12.13. not quite sure if this is bug is directly related to the e.g. non freeing cursors issues? is this somehow related? @CraigglesO can you verify if the cursor issue also got introduced in 5.12.13? |
|
Please include your models @simllll |
|
@CraigglesO Please define what feature is suppose to be along with the |
const userSchema = new Schema({
general: {
email: { type: String, lowercase: true, trim: true },
},
privacy: {
user_register: Date,
user_application: Date,
user_active_sourcing: Date,
company_register: Date,
company_posting: Date,
company_application: Date,
company_club: Date,
company_digital_content: Date
}
});
const DBUserModel = model('UserMongooseBug', userSchema);
await DBUserModel.create({
general: {
email: 'test'
},
privacy: {
user_register: new Date()
}
});
const mailBounceSchema = new Schema(
{
email: {
type: String,
unique: true,
required: true,
dropDups: true
},
amount: { type: Number, default: 0 },
lastError: Schema.Types.Mixed
},
{
timestamps: true
}
);
const DBMailBounceModel = model('MailBounceMongooseBug', mailBounceSchema);
await DBUserModel.create({
email: 'test',
amount: 1,
lastError: {
bla: true
}
}); |
|
IslandRhythms
added
can't reproduce
|
Is this intended that you have added the can't reproduce label @IslandRhythms ? If i run your script with mongoose ^5.13.3: I run with |
|
My bad, missed the version number |
IslandRhythms
added
confirmed-bug
|
So actually looking at it, the mailbounce key does exist but it is null |
|
|
The fact that our outputs don't line up usually means there is something wrong on your end. |
|
|
Exactly, but this is wrong ;-) mailbounce should exists on "selUser", but not under privacy. it is set to the "selUser.mailBoucne" not to "selUser.privacy.mailbounce" |
IslandRhythms
added
the
confirmed-bug
|
Yeah, right.. the script that breaks has a batchSize set to 150000. Do you think setting it smaller would solve this issue? |
|
Yep I think the call stack size exceeded error is due to #9366, which made populate work with I think a better option would be to cap the batch size we use for |
|
Intersesting, but in this case I'm not populating anything. Only going through the documents as they are (I only can't use lean because I need some information about the model), also in this case we are talking about very small documents where a high batchSize actually has a very good performance impact. That said, I'm not 100% sure which query is causing this, but in general we are not using populate (there where only some queries left that used mongoose's populate, that we changed in the meantime too ..), as we have our own implementatoin / cache layer to do population,.. which makes my guess stronger that this is not directly related to this 3rd issue. But it's the cause of the 2nd one pretty sure :) |
|
@simllll in v5.13.6 we'll clear the stack every 1000 docs, so this stack overflow shouldn't happen anymore. Let me know if this issue persists. |
I am experiencing the same issue with mongoose v5.13.22, As @vkarpov15 said that the stack will be cleared after every 1000 docs. In my case I guess the document size is big enough that my stack gets filled up at around 715 documents before I get the RangeError. I think instead of a constant 1000 document count this number should be decided based on the document size. |
|
That's a good suggestion @harshrai654 . Unfortunately Mongoose 5.x is EOL, but we will look into this issue and try to fix for 6.x+ |
vkarpov15
added
enhancement
perf(cursor): clear the stack every time if using populate with batchSize to avoid stack overflows with large docs
Do you want to request a feature or report a bug?
bug
What is the current behavior?
there are some other issues out there with maximum call stack size exceeded, but I#m unsure if they are related to this one:
If the current behavior is a bug, please provide the steps to reproduce.
I cannot really reproduce it, as it seems it only happens from time to time, but it crashes the whole node process after that.
I see following strange behaviours:
The related has this structure:
with one data point set:
after the mongoose result, I add some more data to the object, in this case I added to the root object "user.mailBounces = {...}".
when I use Object.values of user.privacy I get also the mailBounces values returned?
2. For some models and queries it seems if I use populate the call stack size exceeded error appears. It only helps to remove populate from these kind of queries. THe strange thing is: it seems to be related with the data of the collecitno, as this is only happening for us on produciton, but not on the test environment?
3. The error above: The main issue with that is that I cannot even say where it is coming from, as the stacktrace doesn't let me debug this. But I guess it has something similar like point 2.
I put all these 3 issues into one ticket, as I hardly believe this is all related with virtuals somehow?
What is the expected behavior?
no errors, I just recently upgraded from
"mongoose": "5.10.19",
to
"mongoose": "^5.13.2",
..there were no issues with 5.10.19.
What are the versions of Node.js, Mongoose and MongoDB you are using? Note that "latest" is not a version.
mongoose: 5.13.12
node: v16.4.0
mongodb: 4.4.1
The text was updated successfully, but these errors were encountered: