A significant effect of aggregation dynamics for aqueous ferrofluid (AF) and kerosene based ferrofluid (KF) using magnetic field dependent capillary viscosity and magneto-optical relaxation measurements is studied. For better comparison parameters of AF and KF are kept similar. Ferrohydrodynamic equations of chain forming ferrofluids, dilute ferrofluids, and Brownian dynamic simulations are compared. It is observed that the rotational viscosity of AF is larger than that of KF due to field induced aggregates in it and strong dipolar interactions. It is also observed that at Ωτ ~ 0.04 both AF and KF viscosity becomes almost similar, suggesting similar behavior at that shear rate. The magneto-optical relaxation in AF exhibits nonexponential behavior when relaxed from higher magnetic field and follows irreversible thermodynamics, whereas for KF the relaxation is exponential and follows the effective field method. This discrepancy is explained based on aggregation dynamics of magnetic particles. Results are well described by the corresponding theoretical models.