将两个数组与模型合并

So I have a simple table:

messages:
 - id
 - user_id - user that "sent" the message (the author)
 - to_user_id - user that "received" the message
 - body
 - created_at
 - updated_at

I'm using Laravel 5.2, I have a model that looks like this:

class Message extends Model
{
    public function user()
    {
        return $this->belongsTo('App\Models\User'); //This should return only the sender
    }

    public function scopeSentByUser($query, $user_id)
    {
        return $query->where('user_id', '=', $user_id);
    }

    public function scopeReceivedByUser($query, $user_id)
    {
        return $query->orWhere('to_user_id', '=', $user_id);
    }

    public function scopeNewestFirst($query)
    {
        return $query->orderBy('created_at', 'DESC');
    }
}

In my controller I have two variables that call few model methods:

$this->sent = Message::sentByUser(Auth::user()->id)->newestFirst()->get();
$this->received = Message::receivedByUser(Auth::user()->id)->newestFirst()->get();

I was wondering how can I merge them into threads. Basically I don't have anything like threads in my app. Users can write messages between and it's a single thread. Like Zuckerberg's social network, or Google's chat client.

I want to be able to keep them in order (or I can order them after grouping) and have something like:

$threads = [
    [
        'participants' => [1, 2], //those are users' ids
        'messages' => [
            (Model)Message,
            (Model)Message,
            (Model)Message,
            (Model)Message,
            (Model)Message
        ]
    ] 
];

Edit:

Based on the accepted answer I ended up having the following code:

$query = self::where('user_id', $logged_user_id)->orWhere('to_user_id', $logged_user_id)->get();

$inUserId = $query->lists('user_id')->toArray();
$inToUserId = $query->lists('to_user_id')->toArray();

$mergedIds = array_merge($inUserId, $inToUserId);
$uniqueIds = array_unique($mergedIds);

unset($uniqueIds[array_search($logged_user_id, $uniqueIds)]); //Remove logged in user ID

$combinations = [];
foreach ($uniqueIds as $id) {
    $combinations[] = [$id, $logged_user_id];
}
$threads = [];
foreach ($combinations as $key => $combo) {
    $threads[] = [
        'receiver' => $combo[0] == $logged_user_id ? User::find($combo[1]) : User::find($combo[0]),
        'messages' => self::where(function ($query) use ($combo) {
            $query->where('user_id', $combo[0])->where('to_user_id', $combo[1]);
        })->orWhere(function ($query) use ($combo) {
            $query->where('user_id', $combo[1])->where('to_user_id', $combo[0]);
        })->orderBy('created_at', 'ASC')->get()
    ];
}

return $threads;

The differences are: pluck() is replaced with lists()->toArray()

After merging the arrays (array_merge) and picking the unique values only (array_unique) I unset the logged in user's id. Because I don't need it.

Also, removed that method for getting the distinct pairs, since it's not applicable. I believe it should be working now properly.

Will update after some more complex testing is done.

Posted this as update, since I cannot edit answers. And my corrections are too few to post my own answer.