Before I forget, here's the strategy I came up with.

Avaddon aggros and brings back a ton of minibros on his track. Edgar and Luna will pick off some of the minibros that are following after him, as well as some stragglers and any that get onto Artemisia and Blaike. Balder, Ianthe, and Mitra will make sure the herd stays bunched up while killing any that get too close to them. Artemisia and Blaike will work together to create a Fire Vortex, with Artemisia using her wind magic/Aero and Blaike using his black magic/Fire. Once most of the minibros are eliminated, the group will slowly pick off the stragglers.

Afterwards, our choice on exploring the forest I suppose/going deeper into the nest.

For fighting the Mama Malboro, that's a different story. For the most part this is a ranged battle and the group will need to use the bombs Edgar made wisely. Everyone puts the Fire Aldite into their weapons and will zone the Malboro out. Avaddon and Edgar will be the ones choosing where to use the fire bombs, as we will want to knock down trees in an attempt to damage and immobilize the Malboros. Frontliners will be Balder, Ianthe, and Mitra will fight on the frontlines to keep the tentacles away, while Artemisia, Blaike, and Luna will fight from the backlines with Blaike going into the frontlines should anything happen to one of the frontliners. Artemisia and Blaike will prioritize damage, but should also work to burn any tentacles that try to harm, disarm, or disable their allies movements while Luna will focus on shooting at the Malboros eyes.

While nothing will go completely as planned, these are my two ideas I currently have for how to approach combat.