Madrid have fallen to back-to-back home defeats and have lost two of their first four games in the Champions League.
Vinicius Junior’s penalty equalised for Madrid after Malick Thiaw’s opener, but strikes from Alvaro Morata and Tijjani Reijnders secured a deserved victory for Milan.
Madrid have now lost their past two games after being beaten by rivals Barcelona 4-0 in El Clasico, who they now trail by nine points in La Liga.
And Ancelotti’s reigning European champions have only six points from four games in the Champions League, following their defeat at Lille in October.
“We have to be worried, the team’s not giving a good account of itself,” said Ancelotti.
“We have to be more compact, more organised, we’ve let in a lot of goals.
We’re not well organised on the pitch and we have to work on that.”
Milan have also had a mixed start to the season – they only had one win from their first three Champions League games and are seventh in Serie A, eight points behind leaders Napoli.
But an impressive display in Madrid was praised by their manager Paulo Fonseca.
“The players came here with courage, they didn’t have any fear,” the Portuguese coach said.
“In the first half we created plenty and in the second half we were on the back foot, but we fought together.
We definitely deserved to win.”
Milan led from a corner after 12 minutes, centre-back Thiaw heading home Christian Pulisic’s near-post delivery for his first goal in red and black.
Madrid equalised nine minutes later through Vinicius, in his first game since he and the club boycotted the Ballon d’Or ceremony, claiming they were “not respected”.
After the Brazilian winger was tripped in the box by Emerson Royal, Vinicius chipped keeper Mike Maignan from the spot before kissing the Real Madrid badge in celebration in front of ecstatic support.
But in an end-to-end encounter, Madrid lost the ball in midfield six minutes before half-time and failed to respond when Rafael Leao’s low shot on the turn was palmed out by Andriy Lunin.
Former Real and Atletico Madrid striker Morata finished high into the roof of the net.
It was a hugely frustrating evening at times for Madrid fans, who saw Kylian Mbappe – who now has just one goal in six games – continue to struggle in front of goal.
The 25-year-old’s favoured position is on the left wing but Ancelotti has kept Vinicius in that position, forcing Mbappe to play centrally.
As they pushed forward after the interval, Milan nearly punished Madrid on the break – Lunin going full stretch on 55 minutes to brilliantly stop a Leao header.
And the third goal was found 17 minutes from time when Reijnders, who alongside USA international Yunus Musah dominated the midfield all night, burst from deep to smash past Lunin from seven yards.
By contrast Madrid’s stellar midfield endured a tough night.
Jude Bellingham was subbed immediately after the third Milan goal, and booted away a water bottle in frustration as Ancelotti – beaten by the side he won two Champions League titles with before claiming a further three in Madrid – could only watch on.
And a poor evening was capped when a strike by Antonio Rudiger with fewer than 10 minutes to play was disallowed by VAR.
This was Madrid’s first game since being thrashed in the Clasico, due to their visit to Valencia at the weekend being postponed following floods in Spain.
Madrid and Milan wore white T-shirts before kick-off saying “we are all Valencia” both in Spanish and Italian, while there were also banners in the crowd and a giant city flag of Valencia that was displayed during the pre-match minute of silence.
