3D Geometry – Direction Cosines – JEE Main 23 Jan 2026 Shift 1

Solution:

Given equations:
1) $n = 4l + m$
2) $2mn + 10nl + 3lm = 0$

Substitute $n$ from (1) into (2):
$$ 2m(4l+m) + 10l(4l+m) + 3lm = 0 $$

$$ 8lm + 2m^2 + 40l^2 + 10lm + 3lm = 0 $$

$$ 40l^2 + 21lm + 2m^2 = 0 $$

Factorize the homogeneous quadratic:
$$ 40l^2 + 16lm + 5lm + 2m^2 = 0 $$
$$ 8l(5l + 2m) + m(5l + 2m) = 0 $$
$$ (8l + m)(5l + 2m) = 0 $$

Case 1: $m = -8l$
Substitute into $n = 4l + m$:
$n = 4l – 8l = -4l$.
Direction Ratios 1: $(l, -8l, -4l) \equiv (1, -8, -4)$.

Case 2: $2m = -5l \Rightarrow m = -\frac{5}{2}l$
Substitute into $n = 4l + m$:
$n = 4l – \frac{5}{2}l = \frac{3}{2}l$.
Direction Ratios 2: $(l, -2.5l, 1.5l) \equiv (2, -5, 3)$.

Let vectors be $\vec{d_1} = \hat{i} – 8\hat{j} – 4\hat{k}$ and $\vec{d_2} = 2\hat{i} – 5\hat{j} + 3\hat{k}$.
Angle $\theta$ between them:
$$ \cos\theta = \frac{|\vec{d_1} \cdot \vec{d_2}|}{|\vec{d_1}| |\vec{d_2}|} $$

$$ \vec{d_1} \cdot \vec{d_2} = (1)(2) + (-8)(-5) + (-4)(3) = 2 + 40 – 12 = 30 $$

$$ |\vec{d_1}| = \sqrt{1^2 + (-8)^2 + (-4)^2} = \sqrt{1 + 64 + 16} = \sqrt{81} = 9 $$

$$ |\vec{d_2}| = \sqrt{2^2 + (-5)^2 + 3^2} = \sqrt{4 + 25 + 9} = \sqrt{38} $$

$$ \cos\theta = \frac{30}{9\sqrt{38}} = \frac{10}{3\sqrt{38}} $$

Ans. (4)