php - Iterate over Mongo results without running out of memory -
i need find keywords in name/description/tags, etc. of each document , remove them if found. i'm new mongo, i'm following similar script in existing codebase. first, mongocursor
, fields we'll checking:
/** @var mongocursor $products */ $products = $collection->find( ['type' => ['$in' => ['phones', 'tablets']], 'supplier.is_awful' => ['$exists' => true]], ['details.name' => true, 'details.description' => true] );
then, iterate through each document, check each of properties values we're interested in:
/** @var \doctrine\odm\mongodb\documentmanager $manager */ $manager = new manager(); foreach ($products $product) { // find objectionable words in content , remove these documents foreach (["suckysucky's", "deuce", "a z z"] $word) { if (false !== strpos(mb_strtolower($product['details']['name']), $word) || false !== strpos(mb_strtolower($product['details']['description']), $word)) { $object = $manager->find(\app\product::class, $product['_id']); $manager->remove($object); } } } // persist db $manager->flush();
the problem database has hundreds of thousands of records, , looks iterating on mongocursor
, memory usage goes , until runs out:
now @ (0) 20035632 @ (100) 24446048 @ (200) 32190312 @ (300) 36098208 @ (400) 42433656 @ (500) 45204376 @ (600) 50664808 @ (700) 54916888 @ (800) 59847312 @ (900) 65145808 @ (1000) 70764408
is there way me iterate on mongocursor
without running out of memory (i've tried unsetting various objects @ different points, no luck there)? alternatively, query can run directly in mongo? i've looked @ docs, , saw hope in $text
, looks need have index there (i don't), , there can 1 text index per collection.
you don't need fulltext index find substring: right way using regex , return "_id" value, like:
$mongore = new mongoregex("/suckysucky's|deuce|a z z/i") $products = $collection->find( ['type' => ['$in' => ['phones', 'tablets']], 'supplier.is_awful' => ['$exists' => true], '$or': [['details.name' => $mongore], ['details.description' => $mongore]]] ['_id' => true] );
i'm not sure exact php syntax, key inclusive $or filter same mongodb regex on 2 fields.
Comments
Post a Comment